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ABSTRACT 



Freeman, Robert R., Evaluation of the Retri eval of Metallurgical Document 
References Using the Universal Decimal Classificati on in a Computer- 
Based System , Report AIP/UI)C-6 under National Science Foundation Grant 
GN-433» New York, American Institute of Physics, April 1, 1968. 



A set of twenty-five questions were processed against a computer- 
stored file of 9159 document references in the field of ferrous metallurgy , 
representing the 1965 coverage of the Iron and Steel Institute (London) 
information service. A basis for evaluation of system performance charac- 
teristics and analysis of system failures was provided by using questions 
which had previously been processed by the American Society for Metals 
against a data base which contained many of the same documents. The 
Cuadra-Katter model for describing the system evaluation environment was 
used. The results, which were highly satisfactory, led to observations and 
recommendations which contrast the requirements for class definition, 
indexing policy, and search strategy between manual and computer-based 

systems which use UDC. 
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Evaluation of the Retrieval of Metallurgical Document References 
Using UDC as the Index Language in a Computer-Based System 

by 

Robert R. Freeman 

1. Introduction . Documentation of the literature of the field of ferrous 
metallurgy is provided by two English-language services. In the United 
States^ the American Society for Metals (ASM)^ located in Metals Park, Ohio, 
published an abstracting and indexing service, the Review of Metal Literature 
( RML ), covering all fields of metallurgy until 1967.* Bibliographic and 
indexing data were also maintained in a computer-based system and an experi- 
mental information searching service operated for several years. The develop- 
ment of this service, including two indexing languages - the WRU Semantic 
Code and later an alphabetically -arranged natural- language thesaurus - is 
documented in many published articles. Coverage of the ASM service exceeded 

20.000 articles per year. 

The Iron and Steel Institute (ISI) located in London, England, publishes 
a section entitled Abstracts of Current Literature and Book Notices - Iron and 
Steel Institute . Annual author and alphabetic subject indexes are provided. 

ISI also publishes the same abstracts on 3x5 cards, distributing batches 
fortnightly. These cards, unlike the published abstracts, include Universal 
Decimal Classification (UDC) numbers as a guide to the subject content. The 
Abstract and Book Title Card Service, or ABTICS, system has accounted for over 

65.000 abstracts since it began in 1960. The current rate is somewhat over 

9.000 abstracts per year. 

Owing to the greater scope (all metallurgy vs. ferrous metallurgy alone) 
ASM*s system includes a great many of ISI*s 9,000 items each year within its 
coverage. The overlap is not known precisely, but it is estimated to be any- 
where from 5-30% of ASM's total coverage. 

* In January, 1968, Metals Abstracts, combining and superseding RML and 
Metallurgica l Abstracts, commenced publication as a joint service of ASM and 
The Institute of Metals (London). 



- 2 - 









wmmmmm 



The use of UDC in the field of metallurgy is enhanced by the existence 
of the Special Subject Edition for Metallurgy ^, prepared by the C669-Metallurgy 
Committee of the International Federation for Documentation (FID) in 1964. 

Mr, J.P. Saville of ISI, who, with Mr. E. ‘Ohman of FID, first suggested use 
of UDC for a metallurgical documentation service , served as rapporteur of 

the committee. 

When the American Institute of Physics (AIP) commenced its UDC Project 
in 1965, Mr. Morris L. Pearl of ISI suggested that the ISI ABTICS cards would 
provide a useful corpus of document references for a test of the UDC in a 
mechanized retrieval system. ASM subsequently agreed to cooperate in any 
suitable way. The AIP, ASM^ and ISI staffs reached the agreement in 1966 that 
the 1965 coverage of both services would provide a useful basis for a test 
of UDC, according to methods described below. 

2, Objectives . A previous report of the AIP/UDC Project documented the 
demonstration of the fact that UDC may be used as the indexing language in a 
mechanized system^. The objective of the work reported here and in a separate 
report to be published subsequently is to illustrate and analyze the results 
which may be obtained in a test environment which reasonably simulates some 
of the features of a real information retrieval system. 



X TTniv ersal Decimal Classification; Specia l Subject Edition for Metallurgy , 
FID No. 362, ISI Special Report 84, The Iron and Steel Institute, London, 

September 1964. 

2. E. Ohman and J. P. Saville, "The Universal Decimal Classification Applied 
to Metallurgical Literature", Journal o f the Iron and Steel Institu^, 177, 
183-188 (May, 1954). 

3 Robert R. Freeman and Pauline Atherton, File Organization and Search 

S trategy Using the Universal Decimal Classification in Mechanized Reference 
RetriSTOl Systems. Report No, AIP/UDC-5, American Institute of Physics, 

New York, September 15, 1967. National Science Foundation Grant 01-933 
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The purposes behind this objective are as follows: 

(a.) to be able to show which^ if any^ features of the UDC influence the 
performance of the retrieval system^ in either direction; 

(b.) to be able to show which^ if any^ indexing policies used by ISI 
in conjunction with UDC are particularly appropriate or inappropriate^ 
especially with respect to the transition from a manual to a mechanized system. 

(c.) to be able to show idiich^ if any^ search strategies are particularly 
appropriate or inappropriate for use in the system. 

Knowing in advance that a significant number of documents have been 
indexed by two different indexing languages^ one may be tempted to evaluate 
the relative performance of the two. However^ it is our opinion that the 
confounding of the many factors involved in an information retrieval system 
makes such a goal unrealistic for this limited study^ if^ indeed^ it is possible 
at all. Instead^ the procedure of parallel searches^ described below^ has 
been used only for the purpose of discovery of relevant document references 
not retrievable by the UDC-based system. This result^ in turn^ enables an 
analysis of search failures to be conducted. 

3. Methods and Experimental Design . In this section we have attempted to 
describe the factors which are likely to influence the results of the study. 
Throughout the experiment^ there are implicit assumptions that the conditions 
described represent a statistically valid sample drawn from the actual ISI 
system and that static sample files adequately simulate the dynamic real 
conditions. In this study^ we have ne it heir sought to check the veracity of 
these assumptions^ nor have we noted any evidence which leads us to question 
them. 
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3 , 1 , Characteristics of the document collection . As Indicated above ^ the ISI 
ABTICS service had produced a cumulative file of over 65,000 abstracts by the 
end of 1967. The abstracts represent coverage of periodical literature and 
books. The test collection consisted of 9,159 abstracts to periodical articles. 
The abstracts were all published during 1965 by ISI. The form of the abstracts 
Is Illustrated by Figure 1. The file of abstracts Is organized by twenty-three 
fortnightly batches* and by author name within batches, with anonymous papers 
filed under anon . 

FIGURE 1 

SAMPLE OF ISI ABTICS CARD 



KRULIKOVSK/LYA, H.P. 47,300 

LYSAK, L.I. 

y ^ftviiig elanenti on tha ortep tatlon and preoiDltatiOP rata 
ftf the oarblde phase on the break nn of austenite, (YWFftW 

Hfltnlleved. . 1964, (10), 129-135). [in «*»•] 

The introduotlon of 2.7$^ W and 0.45{J Mo Into steel eontftl nin g 14^ Ml 

and 14jS Cr ohangee the orientation of the oryetal lattice vith 

respect to that of the natrix on disintegration of the supersaturated 

-colid solution. The nutual orientation of these lattices has a 

narked Influence on the disintegration process, and the nechanisn of 
this is discussed. - O.A. 

3.2. Characteristics of Indexing records . Each abstract card Included a set 
of one or more UDC numbers, presented, according to standard UDC practice, as 
a chain of numbers related by the colon and plus sign syntactic symbols. Sets 
of unrelated concepts** are allotted separate lines. Only a single order 



669,15*24»26^94i 

6^.111.35* 

669 . 27 /. 20 . 



* the twenty-fourth batch, consisting entirely of book titles, was excluded 
from the sample. 

** l.e. where a document Is concerned with A. and B, separately, not In 
relation to each other. 




imiifi 



Ai 






of urc numbers Is given with each card, the user of a manual system being 
expected to permate the Individual numbers and file according to his own 
Interests. 

Two significant differences exist between the manual ABTICS indexing 
records and the e<iulvalent In the experimental retrieval system. Firsts each 
UDC number In a chain is treated as a separate, individually retrievable 
descriptor, no consideration being given to the syntactic devices. Second, 
the UDC "point -zero-*, "point -double -zero", and "dash" auxiliary forms, which 
appear in normal form to be part of the UDC number to which they are suffixed, 
are also treated as separate descriptors within the machine system. 

The quantitative data given in Figures 2 and 3 reflect the indexing 
records as they appear in the machine system. The reader should recall, 
however, that the 2,921 UDC descriptors used are not equivalent to 2,921 
mutually exclusive classes, but are. Instead extensively and explicitly 
related classes. Figure 4 shows the same data In graphic form. In light 

4 

of the evidence given by Houston and Wall for a variety of other Indexes, 
we should not be surprised to see that the use of UDC ilescrlptors closely 
follows the log-normal distribution. 



4. N. Houston and E. Wall, "The Distribution of Term Usage In Manipulative 
Indexes", American Documentation. 1^(2), 105-114 (April, 1964). 


















FIGURE 2 

Quantitative Characteristics of UDC Indexing Records 

Number of Postings: 28^131 

Number of Documents Indexed: 9^159 

Number of UDC Descriptors Used: 2,921 

28 131 

Average Depth of Indexing = 

Range: 1-8 

28 151 

Average Loading of Descriptors = 2*921 " 

Range: 1-2073 
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FI6UBE 3 

P^t^ bution of Use of UPC Paaorlj atftTO 



Column W - Number of Documents Posted to a Descriptor 

Column X ■ Number of TJDC Descriptors Having the Number 
of Postins Shown in Column W 

Column Y ■ Number of Descriptors Having W or fewer 
Documents Posted (Cumulation of x) 

Column Z - Number of Documents Posted to Descriptors 
Having W or fewer Documents Posted 
(Cumulation of W times X) 



w 


X 


Y 


Z 


1 


1234 


1234 


1234 


2 


421 


1655 


2076 


3 


233 


1888 


2775 


4 


177 


2065 


3483 


5 


100 


2165 


3983 


6 


81 


2246 


4469 


7 


51 


2297 


4826 


8 


65 


2362 


5346 


9 


44 


2406 


5742 


10 


43 


2449 


6172 


11 


27 


2476 


6469 


12 


32 


25O8 


6853 


13 


21 


2529 


7126 


14 


32 


2561 


7574 


15 


16 


2577 


7814 


16 


19 


2596 


8118 


17 


21 


2617 


8475 


18 


14 


2631 


8727 


19 


14 


2645 


8993 


20 


11 


2656 


9213 


21 


18 


2674 


9591 


22 


8 


2682 


9767 


23 


10 


2692 


9997 


24 


10 


2702 


10237 


25 


9 


2711 


10462 


26 


9 


2720 


IO696 


27 


9 


2729 


10939 


28 


11 


2740 


11247 


29 


12 


2752 


11595 


30 


5 


2757 


11745 


31 


8 


2765 


11993 


32 


5 


2770 


12153 


33 


4 


2774 


12285 


34 


8 


2782 


12557 


35 


8 


2790 


12837 


36 


4 


2794 


12981 


37 


5 


2799 


13166 


38 


2 


2801 


13242 



W 


X 


Y 


Z 


39 


3 


2804 


13359 


40 


7 


2811 


13639 


41 


1 


2812 


I368O 


42 


4 


2816 


13848 


43 


7 


2823 


14149 


44 


1 


2824 


14193 


45 


5 


2829 


1441 8 


46 


4 


2833 


14602 


47 


1 


2834 


14649 


48 


3 


2837 


14793 


49 


1 


2838 


14842 


50 


2 


2840 


14942 


51 


1 


2841 


14993 


52 


3 


2844 


15149 


53 


I 


2845 


15202 


54 


2 


2847 


15310 


56 


3 


2850 


15478 


57 


1 


2851 


15535 


59 


1 


2852 


15594 


61 


1 


2853 


15655 


62 


1 


2854 


15717 


63 


2 


2856 


15843 


66 


1 


2857 


15909 


67 


1 


2858 


15976 


72 


2 


2860 


16120 


73 


1 


2861 


16193 


74 


2 


2863 


16341 


75 


1 


2864 


16416 


76 


2 


2866 


16568 


78 


1 


2867 


16646 


80 


3 


2870 


16886 


81 


1 


2871 


16967 


84 


1 


2872 


17051 


86 


1 


2873 


17137 


87 


1 


2874 


17224 


90 


2 


2876 


17404 


93 


1 


2877 


17497 


95 


1 


2878 


17592 



W X I Z 

96 1 2879 17688 
98 1 2880 17786 
100 1 2881 17886 
101 1 2882 17987 
108 2 2884 18203 
109 1 2885 18312 
113 2 2887 18538 
116 2 2889 18770 
118 1 2890 18888 
121 1 2891 19009 

129 1 2892 19138 

130 1 2893 19268 

131 2 2895 19530 
133 2 2897 19796 
139 2 2899 20074 
147 1 2900 20221 
151 1 2901 20372 
155 1 2902 20527 
160 1 2903 20687 
167 1 2904 20854 
169 1 2905 21023 
176 1 2906 21199 

184 1 2907 21383 
218 1 29O8 21601 
221 1 2909 21822 
231 1 2910 22053 
237 1 2911 22290 
264 1 2912 22554 

273 1 2913 22827 
286 1 2914 23113 
391 1 2915 23504 

393 1 2916 23897 
411 1 2917 243O8 
445 1 291 8 24753 
600 1 2919 25353 
705 1 2920 26058 
2073 1 2921 281 31 
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3,3, Characteristics of the Retrieval System and Records , The Combined 
File Search System (CFSS)^a package of programs developed distributed by 
IBK Corporation for the 1401 computer^ was used for the experimental system. 
Actual operation took place on an IBM 360/30 with a 16 k memory and four tape 
drives^ using the 1401 emulation mode. Thorough program documentation Is 
available from IBM Corporation^^ while the use of the system In conjunction 
with UDC Is described elsewhere by the author^. It should be noted that the 
Combined File Search System was also employed by the American Society for 
Metals, 

It was necessary to decide upon how much of the manual ABTICS record 
to convert to machine-readable form. The minimum requirement would have 
been the UDC Indexing and the ABTICS abstract number for each record^ the 
maximum Including In addition the complete bibliographic reference and the 
abstract. The advantage of the minimum requirement would be lower cost both 
In data conversion to machine-readable form and In less computer time used^ 
chiefly In tape handling operations. The disadvantage would be that manual 
retrieval and reproduction of the ABTICS cards would be required In order 
to provide a basis for relevance judgements. Since the CFSS has a rather 
poor report generator from the point of view of ready Intelligibility to the 
untrained user^ we decided that use of the ABTICS cards for relevance judgement 
purposes was desirable. 



5, D,D, Prentice^ G, deGraw, A, Smith, and I, A, Warheit, 1401 Information 
Storage and Retrieval System (The Combined File Search System) . IBM 1401 
General Program Library Number 1401-10,3,047 (Version 2), San Jose, 
California, IBM Corporation, April 21, 1965, 

6, Freeman and Atherton, og, clt . 
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consequently, a minimum machine-readable record seemed to be indicated. 
Since the manual ABTICS file was ordered by author within fortnightly batch, 
it was necessary to devise a record identification code which included these 
items as well as the abstract number. As a check against possible record 
identification code errors, we also keypunched the document titles. 



3 . 4 . Pharacteristics of the Query Set. Me felt that the best approach to 
simulating a real UDC-based information retrieval system would be found in 
the use of real questions, submitted by metallurgists who had no knowledge 
of or concern with the internal characteristics of the system. As explained 
above, we also desired to pose the same queries to the system operated by 
the American Society for Metals as a check on the performance of the OTC-based 



system. 

3.4.1. Source . The American Society for Metals therefore agreed to provide 
twenty.five questions which had been processed against its 1965 files and which, 
in ASM’s judgement, were representative of questions submitted to a real 
metallurgical information service. The number of questions selected was 



purely arbitrary. 

3.4.2. Keeotiation . ASM supplied the project with a record of the question; 
notes added by an analyst who usually negotiated by telephone or letter with 
the source of the question; the encoded form of the question, as processed 
by the ASM system; and a list of Pevlew of Metal Literature (RML) abstract 
numbers produced as a result of the search. No recc'd of either the analyst- s 
judgement of the system output or the source -s judgement of the material 

transmitted to him was obtained. 

The records described in the preceding paragraph wore transmitted to 
ISI in order that appropriate OTC numbers might be assigned for the experimental 




1 
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retrieval system. Since the source of the questions could not be revealed 
by ASM^ no further negotiation would have been possible^ even if it had 
been desirable. 

Mr. J.F. Seville of ISI suggested UDC numbers for each question and 
provided a set of notes outlining how the numbers should be combined into 
logical statements. Mr. Seville directs the assignment of UDC numbers to 
ABTICS cards and^ as described above ^ was a prime mover in the de\:^lopment 
of the Special Subject Edition of UDC for Metallurgy . 

3.4.3. Translation to Logical Statement . Using the formulations thus 
obtained^ the author translated the questions into logical statements in 
the form and format required by the CFSS. An apparent drawback of CFSS was 
that^ although all statements could be expressed successfully^ the lack of 
sufficient capabilities for "nesting" made some statements unnecessarily 
lengthy and time-constiming to set up. For example if a question calls for 

^[A or (B and C)] and (D or E)^ , 
it is necessary to keypunch the equivalent of 



[(A and D) or (A and E) or (B and C and D) or (B and C and E)] 
for CFSS. However^ our experience with the experimental data indicate^ as 
one might guess from observing that the depth of indexing is close to 3^ 
that extremely complex logical statements are generally unnecessary and 
even harmful. Thus the net effect of the drawback is diminished. 

3.4.4. Search Strategy Variations . Part of our aim was to experiment with 
search strategies in conjunction with the UDC. For this reason^ each question 
was encoded and run at least two and often three or even four different ways. 
Search statements generally Involved use of one or a combination of the 
strategies listed below. Xt was impossible to follow a consistent pattern 
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of strategy variation^ owing to the nature of the questions and the 
relationship of the UDC to them. 

3.4.4. 1. Synonym Control . By its nature and purpose^ the UDC number 
generally represents the confounding of synonyms and near -synonyms which 
would have to be accounted for by the searcher in a system with an uncon- 
trolled vocabulary. 

However^ it still remains that UDC^ not being truly a faceted 



creates numbers which must be treated as synonyms with respect to a given 
question^ where natural language provides a single term. Thus "brittleness" 



is represented by 539.56^ where it is thought of as a physical property of 
materials^ while it is also 620.192.49 where it is thought of as a 
structural defect in the testing of materials. Likewise^ the chemical 
elements appear twice^ according to whether the viewpoint is that of 
chemistry or of engineering and metallurgy^ or even four times^ if one 
includes their occurence in minerals and the mining of the minerals. 

A carefully controlled information service will generally avoid the 
potential difficulties of such problems^ particularly if the indexers are 
also involved in search analysis. However^ as we shall show later^ control 
of internal UDC-synonyms and partial synonyms is an important search strategy. 
3.4.4. 2. More Specific. Hierarchically-related Concepts . The first version 
of the search statement for each question involved requests for exact matches 
between UDC numbers in the indexing record and those in the statement. 

However^ it is to be expected that a user who wants to know about open-hearth 
furnaces (669.183.21) should have a chance to see a paper on open?>hearth 
furnace walls (669.183,211.3). CFSS provides for this by permitting the 



classification and being organized by scientific discipline and by industry, 
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analyst to suffix a $ sign to signal that he wants either an exact match 
only or all more specific^ hierarchically-related concepts as well. 

3.4.4. 3. Broader. Hierarchically-related Concepts . If the search analyst 

specifies the use of UDC number which is subsumed under a broader 

concept Xj, then a possible search variation is to request (Xj or Xn$), 
which demands retrieval on an exact match with 

specific topics^ or with the related^ but broader topic. 

3. 4. 4. 4. Filial Concepts . Continuing the preceding example^ if X^ subsumes 

not only X^^, but also ^ 13 ^ ^ 14 ^ ^ 15 ^ theve are two additional strategies 

which have the effect of retrieving documents on potentially related topics. 

The request (X^ or X^^$ or X ^2 ^13 ^14 retrieve all 

covered by 3.4.4. 3. and documents Indexed by filial concepts as well. The 
request (Xj^$) is even broader, being designed to retrieve even documents 
Indexed by concepts hierarchically-related, but more specific than the 
filial concepts. 

3.4.4. 5 . Diminished Logical Restrictions . If the analyst specified (X and Y) 
as the search requirement, two search strategy variations Involve loosening 
the restriction of the logical product. One variation calls Instead for 

(X or Y), the other simply for (X). 

All of these variations as well as others have been suggested by 
Cleverdon and Mills^. Those described here were the set which were empirically 
useful in this experiment. Detailed descriptions of each question, the ASM 
and ISI analysts' notes, UDC numbers, and the various search formulations will 
be found in the Appendix. 



7. C.W. Cleverdon and J. Mills, "The Analysis of Index Language Devices", 
pp. 451-454 in H.P. Luhn (Ed.), Automation and Scientific Communication. 
American Documentation Institute, Washington, 1963. 
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3.4.5. S»"»«"arY of the Query Set . In summary, there were 25 original questions, 
submitted by metallurgists or others interested in metallurgical topics, who 
had no knowledge of the system or that a test was being conducted. The 
search strategy variations produced a total of 64 queries submitted to the 
machine system. These queries included 590 UDC descriptors, of which 510 
were "truncated", i.e. demanded retrieval on either an exact match or on a 
more specific, hierarchically-related term. The UDC descriptors were 
grouped in 196 logical units, each unit consisting of either a single UDC 
class number or an expression of the logical product of several such classes. 



From these figures we can also observe that, on the average, we 
required the logical product of three classes (590/196) in order to satisfy 
the conditions specified. Recalling that the depth of indexing is also 
almost exactly three, we may predict that there will be considerable 
variation in the number of documents retrieved by various search strategies 
for a given question. However, the reader is warned that the distribution 
of UDC descriptors is not even over all of the questions. 



3.5. Relevance Judgements. 



3.5.1. Assembly of the Search Results . The ISI ABTICS abstract numbers 
retrieved by each search strategy were merged into a single list of numbers 
for each search question, a record being kept of which strategies caused 
retrieval of each document. The corresponding abstract cards were pulled 
from the file and xeroxed as "batch-one" for each question. 

The next task was to establish which of the documents were retrieved 
by the ASM system were covered by the ISI ABTICS service in 1965, regardless 
of whether they had been retrieved by the UDC search. Those which were 
retrieved by both searches were, of course, readily identified by comparison 
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of the abstracts published by the two services. The remainder^ l.e. those 

covered by ISI ABTICS^ but not retrieved by UDC^ were Identified by comparing 

the ASM abstracts^ published In Review of Metal Literature with the name 

0 

(l.e. author) Index published by ISI% and subsequently with the ISI ABTICS 
abstract. The ABTICS card abstracts of this latter group (retrieved only 
by ASM) were then assembled and xeroxed as "batch two" for each question. 

The relevance judges were not advised of the meaning of the two batches 
of abstracts delivered for each search. Neither were they given any Indication 
of which strategies retrieved which abstracts^ nor even what the final search 
strategies were. 

3.5.2. The Judging Environment . Probably the most exhaustive list of 
dondltlons affecting relevance judgements has been given by Cuadra and 

9 

Katter and their associates . We shall attempt to characterize the judging 

0 

environment by recourse to this mddel^ to the extent that we have data 
available. The model Indicates that a relevance judgement Is a product of 
the Interaction of characteristics of the document^ the Information requirement 
statement^ the judge^ and the judgement conditions. The judgement Is then 
translated Into a relevance rating value by application of the available mode 
of expression. Many of the Individual characteristics^ discussed In detail 
above^ are only noted again here for the sake of completeness. 



8. Index to Publications of the Iron and Steel Institute. 1965. pp. 1-111, 
London, 1966. 

9. System Development Corporation, Experimental Studies of Relevance 
Judgements! Final Report. Volume 1. Project Summary. Santa Monica, 
California, 30 June 1967. Report TM-3520/001/00. NSF Contract C-424. 
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3. 5.2.1. The Document . 

a. Subject matter . Ferrous metallurgy and closely related areas. 

b. Diversity of content within the do cument. In most cases ^ the 
document upon which the relevance judgement was based was an abstract. 

The Judges were free to consult the full document upon which the abstract 
was based if they chose. Considering both possibilities, the diversity of 
content probably includes a range of quantitative property, process,and 
physical composition data, scientific theory, experimental design, and 
narrative, descriptive, and critical text. Reported data may have been 
generated from laboratory experiments, controlled pilot plant scale operators, 
or full industrial plant scale operations. Although the subject content is 
probably very homogeneous, the factors listed above potentially could have 

had considerable effect upon relevance Judgements. 

c. mfficultv level. While no data are available, we shall assume 



that this factor was not Important. 

Scientific "hardness** of the document . If we adhere strictly to 

Cuadra and Katter's definition^® of ''hardness”, the abstracts used for 
judgement are almost certainly less "hard" than the documents on which they 
were based. Aside from this point, we may note the diversity of content 



among the documents, which ranged from scientific papers, cast in the 
traditional form of physical or chemical research reports; engineering papers 
on plant design, materials testing, or macro-scale operations, for example; 



10 Ibid., p. 34. "The hardness of a particular document is indicated by the 
'Vision of the language and the relationship among the stated aims of 
the document, the conclusions, the methodology of inquiry, and the 
supporting data. If any of these, or the relationship between them, is 
ill-defined, nonexistent, unclear, questionable, or otherwise precarious, 
the document would be considered less 'hard'." 
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trade magazine articles on Industrial news^ market studies^ etc.; and 
reviews of various types. There was some evidence that the Judges applied 
different standards to abstracts of reviews than to "harder" documents. 

e. Amount of "Information" In the document . Assuming that this 
Item refers to the occurrence of unexpected subject matter^ we have no 
data to report. 

f. Level of condensation and textual attributes . The abstracts 
ranged from simple title abstracts (bibliographic references with no 

I accompanying text) to a length of several hundred words. The most typical 

set probably ranged from 50-100 words per abstract. We have no data on the 
I type- token ratio. 

g. Special Qualitative attributes . No data. 

3. 5.2.2. The Information Requirement Statement . 

a. Subject matter . The subject matter was "on target"^ l.e. one could 
probably not conceive of an Information service which was better qualified 
to receive the questions^ considering the subject of Its collection. 

I h. Diversity of content within a question . All questions appeared 

I to be "single" questions^ as opposed to several separate questions embedded 

[ In one. 

I 

I c. Difficulty level. Specificity, and Functional Ambiguity . The range 

j of difficulty was evidenced by the fact that the UDC search analyst encoded 

I many requests essentially without comment^ while In some cases he added 

notes such as 

(1) "this Is both very general and very specific^" 

(2) "This Is remarkably vague^ and the definition does not clarify It" 
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(3) "This is a severe test for any scheme of classification or any 
thesaurus 

In a few cases, the "definition” (notes added by the ASM search analyst 
during negotiation with the source) considerably augmented or modified the 
original statement. The specificity of the questions is indicated by the 
fact that in almost all cases, less than one half of one percent of the file 
was retrieved. See the discussion of "recall" below. 

d. Textual Attributes . No specific data. 

3. 5. 2. 3. The Judges . Cuadra and Katter list the following characteristics 
of Judges in their model: knowledge/experience, intelligence, cognitive 
style, biases. Judging experience, vigilance level. Judgement attitude, 
concept of relevance, use orientation, expectations regarding distributions, 
and error preference. The kind of controlled experimentation required to 
relate each of these items in detail to the present work was beyond the 
scope of the study. Instead, we shall present a condensed description of 
the Judges, referring the reader also to the following section on Judgement 
conditions. 

Since we did not have recourse to the actual sources of the questions, 
we decided to request that representatives of ISI and ASM act as relevance 
Judges. Consequently, what we have are Judgements which lie somewhere 
between those of a system operator and an impartial expert subject specialist. 
The Judges were expert in metallurgical documentation services, but they 
were at least one step removed from the precise mechanics of this set of 
searches. 

Mr. Seville acted as the ISI Judge. It will be recalled that he also 
served as the search analyst and supervised the indexing of the documents 
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by UDC. Although UDC numbers appeared on the ABTICS cards sent to Mr. Seville 
for Judging^ we requested that he disregard them In making his Judgements. 

Five ASM Information analysts took part In the judging^ each acting 
as Judge for several of the questions. Mo question was Judged twice. 

Thus, while all were trained in ASM procedures, we cannot deny the possibility 
of this set of Judgements being confounded by the differing individual 
environmental and personality characteristics of the Judges. 

The ISI Judge did not communicate with the ASM Judges. Therefore, 
overlooking individual differences among the ASM Judgements, we have 
essentially two independent sets of Judgements for each of the twenty-five 
questions. 

The Judges were given identical sets of Instructions. The Instructions, 
described below, were designed to influence the factors of Judgement attitude, 
concept of relevance, use orientation, and error preference. 

3. 5.2.4. Judgement Conditions . 

a. Amount of time permitted . The Judges were instructed to take as 

much time as they felt was necessary. 

b. Order of presentation . The composition of the two batches of 
abstracts presented for each question has been described. The abstracts 
were ordered by author in "batch one" and by the order of occurrence of the 
equivalent abstract in the ASM Review of Metal Literature in "batch two". 

There was no strategy other than clerical ease of assembly behind the use 
of these orders. 

One incident, however, illustrates the effects of this factor. One 
abstract of a document which had been retrieved by both systems was accidentally 
included in both batches submitted to the Judges for the question. In one 
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batchy where the abstract appeared with only a few others^ It was Judged 

1 

relevant by both Judges* In the other batchy where It appeared In the 

midst of a very large group^ almost all of which were Judged Irrelevant^ 

one of the Judges called the same document Irrelevant. | 

c. Size of the document set . See Figure 5 . 

d. Breadth of subject matter In the document set. The subject matter 

I 

was probably quite homogeneous from the point of view of an experienced 
metallurgical documental 1st. It Included ferrous metallurgy^ chemistry^ 

jf 

j 

physics^ chemical engineering^ mining^ mechanical engineering^ and materials 

testing. I 

e. Use of contro l Jud gements . No attempt was made to use control 
Judgements to orient or Influence the Judges. 

j 

f. Social pressure toward convergence . Judgements were made by Judges 
operating alone. The ASM staff who made up one composite Judge may have 

i 

communicated^ but since they were each responsible for a different set of 
questions^ the possibility of pressure toward convergence was minimal. 

i 

g. specification of the task. The Judges were asked to assume their 
normal role as operators of a documentation service for metallurgists. They 
also were asked to assume that 

. 

(1) they were not acquainted with the specific professional or work 
Interests of the question- asker other than those reflected In the question 
Itself and the notes attending the negotiation of the search. 

(2) the questlon-asker Is an experienced research scientist or engineer. \ 

(3) the questlon-asker wishes to review for himself all of the pertinent 
literature prior to commencing work In an area which Is related to his pro- 
fessional competence^ but new to him In many specific details. 

i 
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FIGURE 5 

Size of the Document Set Presented for Relevance Judgement 
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*Batch 1: Retrieved from the ISI ABTICS file by 

the UDC-based retrieval system 

^Batch 2: Covered by both ASM and ISI^ but 

retrieved only by ASM 
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Abstracts with bibliographic reference data were used for relevance judgements, 
UDC numbers appeared with the abstracts^ but judges were requested not to 
allow knowledge of UDC to affect their judgement. Judges were requested to 
refer to the original documents in any cases in which the abstract was too 
brief or otherwise ir adequate to make a judgement. 

h. Definition of relevance . No definition was provided, other than 
that which a judge might infer from the specification of the task. 

3. 5.2. 5. Available Mode of Expression . 

a. Type of scale. Number of Rating Categories, and Kind of Response 
Required . The judges were asked to rate the retrieved documents as 



(1) relevant, (2) marginally relevant, or (3) non-relevant to the question, 
given the task specification. They were provided with a space in which to 
add comments explaining their thoughts or actions. 

b. Availability of anchors . No anchoring stimuli were provided, 

c. Ease of use . The response form was probably close to the 
simplest possible form. 

3,6, Definition and Method of Derivation of Performance Measures , The con- 
founding of many variables discussed in the preceding section serve to 
detract from the value of any quantitative performance measures which might 
be derived. Nevertheless, if one is cautious in attributing overall performance 
to particular factors, even inadequate measures are better than none. 

Three measures have been computed for each question: precision, recall, 
and specificity. The definitions of the former two are well known, while the 
last is simply the ratio of the estimated number of relevant documents in the 
file to the aumber of documents in the file. This ratio provides a measure 
of the degree to which the questions put to the system are of a homogeneous type. 
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The reader is now specifically warned of the difference between the 
measures of performance reported here and those variously reported elsewhere. 
The first difference Is that which results from the relationship of the judges 
to the system. As explained above^ the judges were not the sources of the 
questions and they were not precisely either system operators (for this 
system) or dispassionate subject matter experts. However^ the output of the 
relevance judgement process^ a set of supposed relevant documents^ corresponds 
most closely to what Is commonly forwarded to the source or user of a real 
Information service. Therefore we shall think of relevance henceforth as 
system-operator-relevance . 

The second difference results from the fact that the given number of 
relevant documents in the file is an Inferred estimate^ as described below. 
Consequently^ the performance measures labelled recall and specificity are 
also estimates. 

The reader may recall the theory which states that there is no valid 
manner of categorizing documents as relevant or non-relevant with respect to 
a given question; that Instead^ each document in the file has an objective 
and algorithmically computable relevance to the question. Without denying 
the possible philosophical truth of this theory^ we found it practical to 
accept the more rigid three-way categorization of the human judges' decisions. 
This tactic serves to further differentiate the measures of performance of 
this study from those reported by some other authors. 

Estimating the number of relevant documents in the file for a given 
question is generally the most difficult part of deriving performance figures. 
Our original intent was that a reasonably reliable estimate could be obtained 
directly from the results of the parallel searches of the ISI and ASM files. 

i 
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Consider Figure 6 . The critical figure is the sum of and 
i.e. the set of relevant documents not retrieved by either the UDC-based 
system or by the ASM system, but covered by the ISI ABTICS service. If 
we can safely assume that this set is small enough to be considered negligible, 
then recall is easily computed as 

^ \ ^ = _is_ 

Whether or not this assumption is a safe one depends on the extent to \diich 
the two systems retrieve the same set of relevant documents from among the 
set covered by both services. That is, we looked for A^^ to be large relative 
to either or D . Unfortunately, for the present set of questions, this 
condition did not hold true. With respect to the set of documents covered by 
both services, the two systems acted to a surprising extent as complements 
of each other. 

As a result, it became reasonable to suspect that and possibly F^^ 
were not negligible as originally supposed. Our most reliable estimate of 
recall under these circumstances turned out to be the ratio 

The best estimate of the total number of relevant documents in the ISI ABTICS 
file is therefore 

v<^ " V ■ 

A statistical explanation of this estimate is offered in Appendix B. The 
probabalistic nature of the recall and specificity figures should now be 
clear to the reader. 
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FIGURE 6 



Relationship of Documents Retrieved by ASM and UDC Systems 
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4. Experimental Data , This section includes a complete summary of each 
question and the data generated in relation to it^ while the following section 
concentrates on synthesis of the results and generalization of some of the 
characteristics of the system of interest to our objectives. 

The data are presented in the following format^ where X represents 
the question numbers^ 1-25: 

4.x. 1. Q^iestion 

4.x. l.l. Statement in natural language^ as received by ASM. 

4.x. 1.2. Notes added by the ASM search analyst during negotiation with 
the user. 

4.x. 1.3. Notes added by the ISl search analyst during UDC encoding. 

4.x. 2. UDC descriptors chosen^ including their frequency of use in the 
document file. 

4.x. 3. Encoded logical statements of question. As described above^ usually 
■two or more statements were formulated for each question and run 
separately. 

4.x. 4. Results and Analysis. 

4.x. 4.1. Relevance judgements made by ASM and ISl judges on the composite 
output of all searches run for a given question. Judge 1 was the 
ASM judge and Judge 2 , the 1ST judge. 

4.x. 4. 2. Derived performance characteristics^ Including estimated recall^ 
estimated number of relevant documents in the ISl ABTICS file^ 
precision^ and estimated specificity. All of these measures are 
defined and discussed in section 3.6. 

4.x. 5. Failure analysis. In many cases^ the judges disagreed in making 

relevance judgements. Therefore^ a "failure" is defined as a judgement 
by at least one judge that 















- 25 - 



4.x. 5.1. (1) a document retrieved by ASM and covered by ISI, but not retrieved 

by UDC^ is relevant to the question (the definition does not include 
marginally relevant); or 

4.x. 5. 2. (2) a document retrieved by UDC is non-relevant to the question^ 

i.e. was judged to be in the non-relevant category of the three 
categories available. 



Note: For the convenience of readers who may wish to skip the detailed 
experimental data, section 5 follows immediately. Section 4 is 
continued following page 64. 
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Section 4 continues following page 64 
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5. Analysis and Sumnary . 

5.1. System Qperatina Characteristics . Before proceeding to consider the 
data on system performance reported here^ the reader should review the 
definitions of these measures as applied to this report. Specific data for 
each question are given In section 4 and a tabular summary of the data used 
for computation of the performance measures Is given In Appendix II. 
g^^^^^^^^^^^^ 2 j*o££^g<j_j[>ui||^er__of_referet^^ r etrieved . Twenty* five questions 
produced a total of 707 document references retrieved. The range was 0-218, 
the mean, 28.3 (standard deviation, 53.2) and the median, 8. With so few 
observations and the knowledge that the system Is an experimental one. It 
doea not seem worthwhile to attempt to arrive at a predictive probability 
distribution for the system. 

We can conclude that the system probably would not produce so many 
references on the average as to be a burden on search analysts whose Job 
It would be to filter the output for the user who posed the question. 
However, If the file were ten times as large, a not uncommon size for many 
real systems. It could become necessary to devise more refined and 
sophisticated techniques. 

5.1.2. Precision . At least one document reference was retrieved In 20 
of 25 questions. The overall ratio of relevant documents retrieved to 
total retrieved Is shown In Figur e 7 . 

5.1.3. Estimated recall . The method used for estimating recall required 
that at least one document reference be retrieved In comnon by both the 
UDC system and the ASM system. On this basis, data were obtained for 12 
of the 25 questions. The overall estimated recall ratios are shown In 

Figure 8 . 



» 


















5.1.4. Estimated number of relevant documents. Have made an estimate of 



recall^ based on the retrieval of a common set of documents by both systems^ 
we then divided this figure Into the number of relevant documents retrieved 
by the UDC system to obtain an estimate of the total relevant documents In 
the file. Whereas many Information system tests have relied upon direct 
observation of the number of relevant documents^ the present technique might 
be termed the "Indirect method". For the same 12 questions^ the overall 
average estimated number of relevant documents In shown In Figure 9 . 

5.1.5. Estimated Specificity . This measure^ the estimated number of 
relevant documents divided by the file slze^ 9159^ provides a way of 
predicting the percentage of the file which will be relevant to questions 
posed to the system. On the basis of the limited data^ summarized In 
Figure 10 . we can predict that^ 99% of the tlme^ no more than 1. 15% of 
the file will be relevant to a given question^ with an additional 2.4% 
marginally relevant. 

5.1.6. Validity and Utility of the Measures . The complexity of the 
Cuadra-Katter model alone Is enough to make one question the Idea of 
assigning performance measures. Yet It Is our belief that system managers 
are better off with measures that are not precisely accurate^ but Instead 
Indicate ranges and magnitudes^ than with none at all. As predictors of 
system performance^ the measures given above probably suffer from too small 
a base. They are more valuable as an exercise In methodology of evaluation. 
We encourage other Investigators to repeat or extend the experiments. 

Limited though they are^ the data are the firsts to our knowledge^ to 
result from an application of UDC In a mechanized retrieval system. As such^ 
the measures of performance should lend encouragement to those who are 
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Interested In similar applications. There is little room for doubting 
whether reasonable levels of performance can be attained. 

Before passing on to a summary of some of the factors affecting 
performance^ we shall make one further note of the relevance Judgement 
situation. The values for estimated recall (and consequently those for 
estimated relevant documents and estimated specificity) of relevant documents 
led us to suspect that there was a significant difference In the Judgement 
criteria applied by the two Judges. Several tests were applied In an attempt 
to support or reject this Inference: 

(a.) A variance ratio test using Snedecor's F showed that the variances 
between samples (l.e.^ between Judges) were sufficiently alike to warrant 
the assumption that they were estimates of the same population variance. 

(b. ) Application of the "Student's t" test led to the conclusion that the 
difference between the mean values of the two Judges' results was significant > 
l.e. that a difference of the magnitude observed (0.295 vs. 0.668) would be 
likely to occur only about once in a hundred observations. 

(c. ) An analysis of variance showed that the variance attributable to the 
Judges was not necessarily significant^ the chance of observing such values 
being somewhat greater than 5%. 

We were thus led to no definite conclusion. Possibly an analysis of 
variance which takes Into account the interaction of the Judges with particular 
questions or groups of questions^ such as those of high specificity vs. 
those of low specificity^ would help to explain the differences observed. 

We also note that^ when the relevant and marginal categories are combined^ 
there is no difference of any significance between the two samples. This 
may indicate that^ in the absence of explicit instructions for differentiating 
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"relevant*' from "marginally relevant"^ the Judges applied slightly differing 
criteria. 

Whatever the case may be^ the data from this first test can be of only 
limited value. We are more Interested In discovering why *:he system performed 
as It did than precisely how well It performed. 
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figure 7 
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figure 8 
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FIGURE 9 

Average Estimated Number of Relevant Documents per Question 
as a System Operating Characteristic 
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FIGURE 10 

Average Estimated Specificity Ratios as System Operating Characteristics 

Relevance Categories 
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5.2. Failures . Although it Is useful to have some notion of how 

well a system is performing, the goal of evaluation must lead to an analysis 
of why it performs that way. One might well conceive of "success analysis" 
as a tactic for evaluation. Presumably there would be suitable content- 
analytic or semantic -closeness measures which would reflect the degree to 
which th-2 system accomodated Itself to the user's intention, or, less desirable, 
vice-versa. However, this type of analysis, if not beyond the scope of 
behavioral science methodology, has at least not yet been clearly integrated 
with the evaluation of information system performance. 

Consequently, we shall confine our analysis to the negative aspect, 
viz . , why didn' t the system do what we intended perfectly. While this type 
of analysis is more readily made, the number of instances chalked up to 
questionable judgements serves to remind us of the subtleties of the judging 
process, even where the judges are more-or-less Impartial experts. 

Although the technique of failure analysis is not new, we attempted 
to let the reasons for failure establish themselves, rather than to work 



from a predetermined list. Thus, while there may well be other possible 
sources of error in information systems those reported here may all be 



observed by the reader. 



However, there are two important points to be kept in mind while reviewing 
the failures summarized here. First, a given failure frequently has a 
mixture of interrelated causes. It often appears reasonable to say that, 
if the structure of the indexing language had been slightly altered, the 
Indexer or search analyst would have acted differently. The components of 
a real information system are not the independent and rational units of an 
abstract model. The reader should henceforth remember that a statement 



that a failure resulted from causes a, b, and c is a matter of probability. 
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not certainty. 

The other point deals with assessment of blame for failures. The 
terms "error" and "failure" should be regarded as convenient shorthand 
throughout this discussion. The reader should recall that all of the 
experience > tralnlng> and policies which affected the indexing and search 
analysis processes resulted from a manual information system which was 

created and operated without any notion of a mechanized system. Indeed, | 

i 

this statement applies to the UDC itself. To say that a failure resulted 
from an indexing error overlooks the likely fact that the indexing was 
correctly done according to the policies established for a manual system 
with a given Intended use. 

Having stated these warnings, \ie may now proceed to outline the classes 
of failures. The causes of failure were grouped into five broad classes, 
which include (a.) the indexing process, (b.) the search analysis process, 

(c.) the machine system process, (d.) the UDC, and (e.) the judgement process. 

The first three contain a further element of subdivision, that of whether 
the process involved intellectual decisions or was programmed or clerical 
in nature. Since the project under whi:;h the present work was performed is 
basically interested in the UDC, we shall consider It first. 

5.2.1. UDC Structure or Class Definition . Little, if anything at all, can 
be added here to the sizeable body of knowledge and criticism built up 
over many years with respect to the design of information languages and 
the UDC in particular. This being among the first reports of use of UDC 
in a mechanized retrieval system, it can serve to demonstrate that the 
effects of known types of problems are magnified by the formality of a 
mechanized system. The rapid, adaptive, heuristic characteristics of 
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manual search provide a "forgiveness" factor which Is not present when one 
Is required to commit himself to a given strategy and accept the results. 
The fact that some characteristic of the UDC was at least partially the 
cause of some failures In 40% of the questions In this experiment points 
to the need for attention to this point. 

5.2. 1.1. First-level problems . On the surface^ many observed failures 
can be laid to the existence of partial synonyms^ vaguely defined classes^ 
ambiguity^ lack of specificity^ and "convenience classes". By lack of 
specificity Is meant the existence of a class which Is actually no more 
than the logical sum of several distinct concepts which have not been 
allotted separate UDC numbers. Thus a document on "a" Is Indexed by UDC 
number "(a there being no separate "a"^ "b"^ and "c" available. 



and Is retrieved erroneously In response to a search for concept "c". 



In the case of "convenience classes", "(a + b + c)" exists as a class 
which subsumes "a", "b", and "c", but the sum has no nominal or functional 
Identification distinct from the three subclasses taken together. 



Both of these latter types of problems, lack of specificity and 
"convenience classes" are undoubtedly reflections of the fact that UDC 



has been developed with manual systems In mind. If a document deals with 
several distinct, but closely related topics, fewer cards need be made 
and filed If a single unifying class Is defined. Provided that there are 



relatively x'ew documents which deal with only one of the several topics 
subsumed by a class, the Irrelevant material Is quickly filtered out by a 
manual search. In the computer-based system, however, the file will be 
large and a search for more specific hierarchically-related classes Is easy 



to accomplish. In sum, while the Indexing policy for a manual system may 
















be "use the general class which subsumes closely related topics"^ the 
policy for a mechanized system would be "use the most specific classes". 

These contrasting policies would be reflected In the development of the 
UDC for various uses. At the very leasts the highly specific classes 
should be available for use where needed^ even If they are not needed In 
all systems. 

5.2. 1.2. Second-level problems . However, it is not only the existence of 
specific classes (and associated policies) which affects the performance 
of UDC In mechanized retrieval systems, but the theory according to which 
the classes are defined and the rules governing the statement of relationship 
of classes which are the subjects of a given document. This more basic 
level accounts for the flrst>level problems of partial synonymy, vagueness, 
and ambiguity. We shall borrow the terms used by Gardln to describe these 
two aspects of an Information language, paradigmatic and syntagmatlc 
relationships respectively. 

5.2. 1.2.1. Paradigmatic relationships . The vocabulary of metallurgy and 
related areas of engineering, science, and technology have been widely 
discussed In terms of facet analysis, role Indicators, semantic codes, and 
general categories. An approximate listing of the general categories 
revealed by the file used In the present experiment and by the Special 
Subject Edition of UDC for Metallurgy might be 

a. Material (element, mineral, manufactured product, etc.) 

b. Process 

(1) natural 

(2) man-operated 

c. Machine, apparatus, device 
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d. Purpose^ intended use 

e. Attribute 

(1) physical state 

(2) formally-defined (measurable) property 

(3) form^ shape 

(4) position^ location 

The UDC, as pointed out in an earlier report^^, provides a limited 
facet structure^ consisting of tlme^ place^ ethnic group^ form of document^ 
language of document^ and general subject categories. As a concession to 
the need for further facets^ a general polnt-of-vlew auxiliary and tvo 
special auxiliaries are provided^ the latter two being developed separately 
within major disciplines and technologies as needed. 

In the UDC^ metallurgy and related areas^ particularly materials 
testing and mechanical engineering^ are reasonably good examples of the 
attempt to use the limited tools available for expression of general 
categories. However^ owing to the lack of sufficient facet indicators^ 
the lack of relational operators^ discussed below^ and perhaps also to 
the desire for simple UDC notation^ there remain many examples of hierarchies 
derived by mixing of concepts from different general categories or facets. 

For example, if we have machine "a", material "b", and attribute "c", we 
might form a UDC class "abc" which represents the use of "a" to produce "b" 



11 Freeman and Atherton, 0 £. cit . 



which has attribute "c". Now if we have a request for everything on 
material "b", there are at least two classes which must be searched, "b" 
and "abc". The present experiment revealed several cases in which even 
an expert in the use of UDC in metallurgy did not recall all of the 
possible alternatives under which relevant material was indexed. The 
problem is familiar to classificationists as "distributive relatives". 

5.2. 1.2.2. Syntagmatic Relationships . The capability to specify the 
relationship of two or more concepts which are the subjects of a given 
document is virtually non-existent in the UDC. The existence of a 
relationship is stated by the use of the colon symbol, but no indication 
is conveyed as to the nature of the relationship. 

The consequence of this lack is the need to form classes of the type 

discussed above, which are logical products of classes taken from two or 

more facets or general categories, i.e., to make a pre -coordinated 

classification. Thus, a set of relational operators of the degree of 

12 

completeness suggested by Perreault would not only allow more definite 
specification of syntagmatic relationships among existing classes with 
respect to a given document. It would also provide for simplification of 
the UDC by reduction of the number of classes. However, as we move away 
from a large number of highly pre -coordinated classes toward a smaller 
number of less complex classes, a greater number of classes would be needed 
to express the contents of a document. 

Therein lies a troublesome point for the classification which is 

intended for use in both manual and mechanized systems. In manual systems, 

12 J. Perreault, "Categories and Relators: A New Schema", Revue Internationale 
de la Documentation , 32 , 136-144 (1965) and Emendations to the Relator Schema 
(January, 1968?), available from the author. School of Library and Information 
Services, University of Maryland, College Park, Maryland. 
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brief notations are considered desirable. Even In Its present seml-faceted 
conditions UDC Is sometimes criticized because of the number of characters 
used In classifying an average document. In a computer-based system, 
while the length of the notation is of less concern, within reasonable 
limits, the search analyst's job becomes much easier If the classes are 
not highly pre -coordinated. 

We can conclude, then, that the testing of a system which employs 
a set of relational operators such as Perreault's seems to be In order as 
a next step In exploring use of UDC In mechanized retrieval systems. 

However, we can somewhat pessimistically predict that the full benefit 
cannot be derived therefrom without further revision and systematization 
of the UDC structure. 

5.2.2. System failures attributable to the Indexing process . 

5. 2. 2.1. Failure to Index a concept which appears In the document abstract . 
This source of failure was the most frequently occurring failure^ affecting 
some of the results In 17 out of 25 questions. However^ we must again 
recall that the Indexing was done according to policies devised for a 
manual card Index system. The situation Is quite analogous to some major 
mechanized systems. In which a limited set of the most Important Indexing 
terms Is chosen for use In a published Indexj while a more comprehensive 
set Is stored for use In the mechanized retrieval system. In the present 
case^ the ISI Indexing^ done for a manual system^ could be augmented readily 
(at a greater cost^ of course) for a computer-based system. 

It must be admitted that the criterion for assessing an Indexing failure, 
existence of a concept In an abstract which Is not represented In the Indexing 
record, Is not Infallible. However, the abstract and the indexing record 
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are not independent^ but represent two degrees of compression and selection 
of the information contained in the original document. In fact, many 
information systems combine the two by constructing an abstract in more-or- 
less telegraphic style from a set of indexing terms . 

In a few cases it was possible to infer that the problems of UDC 
structure and notation, discussed above^ may have been a direct cause of 
indexing failures. The -1/-9 auxiliaries, for example serve to indicate 
(1) state and form of metals (-1/-4) , (2) details of machines and apparatus 
(-5/-S) , and (3) process characteristics^ variables , and control conditions 
(-9). The .0 auxiliaries mix properties^ processes) and equipment. Now, 
if a document deals with corrosion-resistant (.018.8), creep-resistant 
(.018.44) steel (669.14) strip (-418) and wire (-426) a number such as 
669.14-418-426.018.44.018.8 might be called for. However, the complexity 
and possible ambiguity of such a notation may work to intimidate an indexer 
into selecting a less specific number or not indexing one or more concepts. 

Assuming the formulation of new indexing policies consistent with the 
requirement of deeper indexing for a mej^hanized retrieval system, indexer 
performance might be enhanced by a special worksheet, which would encourage 
explicit recognition of the facets of each topic in a document. A sample 
design of such a worksheet is given in the following section. 

5. 2. 2. 2. Failure to be as specific as possible . Once again, this type of 
failure is probably indicative of the policy adopted for use in a manual 
system. 

5. 2. 2. 3. Wrong concent indexed . This minor source of failure seems to 
reflect genuine errors.- The cause may lie in the transfer of natural language 
terminology to UDC codes. One tactic for overcoming such errors might be 
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to have Indexers write controlled natural language terms and have the 
computer system translate the terms into equivalent UDC numbers. The 
meaning of the present highly pre-coordlnated UDC classes often requires 
so many words to express that this procedure could be too tedious and 
error-prone to be worthwhile. 

5.2 2.4. Transcription of indexing record . This source of error is common 
to most information systems. The present computer system provides a program 
for automatically checking all input against a table of legitimate codes, 
but this program was not used. Transpositions and elisions, probably the 
most common clerical errors, are easy to detect in a natural language des- 
criptor system. In a digital code system, such as UDC, such errors might 
result in illegitimate codes which would be caught by a dictionary look-up 
procedure, but other errors might well result in legitimate, but wrong codes 
which would not be caught. The fact that four errors of this t}rpe were 
discovered in only approximately 200 documents analyzed (ca. 600 UDC numbers 
out of over 28,000 in the file) for errors leads one to the conclusion that 
the clerical error rate should be investigated more thoroughly. 

5.2 3. System failures attributable to the question analysis and search 
formulation process . 

5.2 3.1. Failure to specify a UDC number which expresses a concent in the 
question . More than any other cause of failure^ this is an indicator of the 
complexity of the UDC structure. The search analyst, as stated above, was 
probably better qualified by experience to suggest UDC numbers to represent 
the search concepts than any other person. Yet in eight out of 25 questions, 
there were additional UDC numbers which could have been used to retrieve 
relevant documents. 







5.2.3 2. UPC mimbera too specific . In this rather minor cause of failure, 
it appeared that the indexing accurately reflected the document content, 
but the search analyst suggested that a more specific, hierarchically 
related UDC number be used for the search. To some extent, ambiguities 
in the definition of UDC classes may have been the cause. 

5.2 3.3. UDC numbers too general . Only one case of this type of failure 
was observed, possibly being attributable to ambiguities in class definition. 
5. 2. 3. 4. Non-ontimum search logic . The use of logical product, sum, or 
difference requirements as part of the statement of conditions for satisfying 
a search request is common to most present-day information systems. As 
shown by Rolling^^, certain operating characteristics of an information 
system can be observed over a period of time to yield reasonably accurate 
tools for predicting the number of documents that will be retrieved by any 
probable search formulation. As indicated in section 3 of this paper, the 
searches were formulated and run without even knowledge of the frequency of 
use of the various UDC numbers in the indexing records. Consequently, it 
was not possible to predict the effects of the tactics used. 

5. 2. 3.4.1. Logical statement too tight . In four cases, the system failed 
to retrieve relevant documents because the search statement required more 
concepts to be present to satisfy the question than were actually necessary. 

5. 2. 3.4. 2. Logical atatement too loose . Eleven searches were affected to 
some extent by the tactics of removing some of the restrictions on the 
statement suggested by the search analyst. It should be recalled that the 




13 Loll N. Rolling, "A Computer-Aided Information Service for Nuclear 
Science and Technology”, Journal o f Documentation, 22(2) , 93-115 (June, 19 ). 
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results of several searches for each question were merged Into a single 

set of abstracts presented to the judges. Knowledge of the system 
operating characteristics would have helped to prevent some of these 
difficulties. 

5. 2. 3. 5. Transcription of search record . Errors of this type are predictable 
and appear to have been minimized. 

5.2.4, Machine system design problems . 

5. 2.4.1. Inadequate processing of UDC Notation . This type of failure was 
anticipated* The UDC provides a notatlonal device for synthesizing special 
class numbers^ thereby avoiding an even greater excess of pre -coordinated 
I classes. If A.l and A. 2 are ^o classes subsumed by A, then by the notation 

A.1'2 we can represent a third class which Is semantically related to the 
former two, but has distinct properties of Its own which are not Inherent 
In the former two. Thus, 669.15 represents alloy steels In general, 669.24 
I represents nickel, and 669.26 represents chromium. By notatlonal synthesis, 

669.15 '24' 26 represents nickel -chromium alloy steels. 

The difficulty Is easily seen. Assume that we have a request for documents 



I about chromium alloy steels, 669.15'26. The present system. In which 










However, both of these transformations Introduce ambiguities which may 
result In retrieval errors. What Is needed Is the ability to request 
669.15x*26, where x represents any string of characters, Including a null 
string. Although this capability could be programmed, we did not wish to 
alter the existing programs of the Combined File Search System for this 
experiment. 

5. 2.4. 2. False coordination of UPC auxiliaries with main class numbers . 

The UDC special auxiliaries, represented by .01. . ./.09. . . and -1/-9 were 
treated as Independent descriptors In the experimental system, even though 
they are frequently dependent upon the main class number to which they are 
appended for their meaning. Consequently, It was possible for a document 
Indexed by X.052:Y to be retrieved by a request for X:Y.052, the result 
being retrieval of a document which Is wildly Irrelevant to the question. 

Our data Indicate that such false coordinations are probably a trivial problem. 



but further work using larger files or files with different Indexing 
characteristics could conceivably produce a different conclusion. 



5. 2. 4. 3. Error during file creation . There are many possible causes for 
the computer system rejecting Input. An operational Information system would 
anticipate the problem and provide for entry of corrections. 

5. 2. 4. 4. Unexplained machine search failures . Some difficulty was encountered, 
apparently with requests which Involve only a single term or a logical sum 
0 xpresslon. In some cases, some of the appropriate documents were retrieved 

and others rejected. 
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S.2.S. Uneupported lnterpretation « iwf ^renoes by .iudaea. Questionable 
.IttdaementB . Many examples of Judgemente which could only he explained by 
infexeneee made by the Judges on the basis of their interpretations of the 
questions and their expert knowledge of metallurgy were in evidenoe. She 
situation was particularly true in the oaee of doouments retrieved only 
by the ASM eyetemt but covered by the ISI iBTICS service. In 10 of the 
23 questions at least one Judge deoided that some members of this set of 
documents were relevant, althou^^ neither the indexing record or the exact 
words of the document abetraot confirmed the judgement directly. 

The frequency of oocurrence of this situation has serious implications 
for those who design or set indexing policy for information systems. 
However, one oan predict that such questionable judgements will always 
occur to some extent, especially where intermediaries are interposed 
between system and user. 

The possibility that idiat Cuadra and Katter have called "social 
pressure toward convergence" may have been a factor should not be dismissed 
If a judge is presented with a set of documents which he knows has been 
retrieved by at least one of two systems, perhaps there is some subtle 
pressure to agree that the documents are relevant. 
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FIGURE 11 



SUMM/UIY OF SYSTEM FAILURES 

[Data should not be Interpreted without reference 
to the accompanying discussion] 



Source of Failure 

(one failure may have several sources) 



1. UDC structure or class definition 



2. Indexing process 

a. Failure to Index a concept which 
appears In the document abstract 

b. Failure to be as specific as 
possible 

c. Wrong concept Indexed 

d. Transcription of Indexing 
record 

3. Search formulation process 

a. Failure to specify a UDC number 
which expresses a concept In the 
question 

b. Too specific 

c. Too general 

d. Logical statement too tight 

e. Logical statement too loose 

f. Transcription of search record 

4. Machine system design problems 

a. UDC notation not adequately 

handled by present system 

b. False coordination of UDC 
auxiliaries with main class 
numbers 

c. Document record apparently re- 
jected from file during file 
creation 

d. Unexplained machine search failures 



Questions In Which this Source Was a 
Cause of Failure (Approximate number of 
affected Items given In parentheses, 
otherwise 1 Is Implied) 

1(3), 6(2), 7(9), 8, 9, 14(3), 20(7), 22, 
24(4), 25(5) 



1(2), 2(2), 6(2), 8(3), 9(16), 10(2), 11, 
12, 13, 14(5), 16(4), 20, 21(2), 22(2), 
23(3), 24(5), 25(2) 

14(5), 15(2) 

2(2), 17 

3, 14, 15, 23 



1, 8, 9, 14(6), 16, 18(4), 20(8), 22 



1, 2(2), 17 
7(3) 

7, 8, 9, 10 

3(2), 4(19), 7(5), 8(8), 15(21), 18(165), 
19(37), 21(3), 22(32), 23(6), 25 
15 



9(3), 25(2) 
7(2), 24(1) 



4 



15(28), 16(20), 18(151), 22(2), 23 



5. Unsupported interpretations or 

Inferences by judges. Questionable 
judgements. 

a. Judgements of & "evance 

b. Judgements of non-relevance 



1, 5(4), 8(2), 9(5), 13, 15(3), 16(4), 

17(7), 18(4), 23 

1, 2(2), 7(3), 15(13), 24(2) 
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6. Conclusions and Recommendations . On the basis of a limited experiment 
in a test environment which reasonably simulates a real Information system, 
we feel justified in encouraging those who wish to make use of UDC as the 
indexing language in a computer-based retrieval system. To the extent 
that the observed results are reliable, valid, and Indicative, the operating 
characteristics of the experimental system are surprisingly good, especially 
when one recalls that the indexing was done with a purely manual system 
in mind. 

The results, particularly the failure analyses, revealed some points 
which should be seriously considered by system designers and managers who 
Intend to use UDC as the indexing language In their system. These points 
may be grouped as (1) search strategies and predictive tools, (2) hierarchical 
searching, (3) new indexing policies, and (4) revisions and Innovations in 
the UDC. 

6.1. Search strategies and predictive tools . It is doubtful whether many 
manual Information retrieval systems have kept precise data on the frequency 
of use of the set of terms available for indexing. Consequently, searching 
a manual system involves only selection of all relevant terms and formulation 
of a statement of the question using the terms and appropriate logical 
operators. 

However, such frequency data are readily available from a computer-based 

system. For the present experiment, we have shown that the frequency of use 

of Individual UDC numbers follows the log-normal distribution which was 

suggested as a common characteristic of information systems by Houston and 
14 

Wall . That is, a very high percentage of the UDC numbers are used to 
index only a few documents each. 

14 Houston and Wall, 0 £. cit . 
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Now, it is well known that the use of the logical product ("and”) 
and difference ("but not") operators has the effect of increasing the 
precision level of a system at the possible risk of a decreased level of 
recall of relevant documents. But, possibly a system operator is willing to 
adopt a policy that he will not be concerned with the precision level (as 
he sees it, distinct from what is passed on to the user) provided that the 

number of documents retrieved can be predicted to be small, e.g.^ 20. 

Frequency-of-use data for the UDC numbers will often show that, if we 
accept such a policy, many searches can be run without the use of logical 
product or difference restrictions. We may expect maximum recall of relevant 
documents and, even if the precision level is low, the total number of 
documents to be examined gives no cause for concern. 

In many cases, it will be necessary to use logical product or difference 
restrictions in a search. Although this precludes direct calculation of the 
expected number of documents to be retrieved, the system operator can collect 
sufficient data from actual searches to be able to predict the number to be 
retrieved in a probabilistic fashion. Rolling has demonstrated this technique 
in connection with the Euratom system. 

To summarize the point, search strategy in a computer-based retrieval 
system using UDC needs to take into consideration actual distributions and 
combinatorial probabilities of the UDC numbers used for indexing. Use of 
such feedback from the system should be a considerable aid to the system 
operator in attempting to optimize the balance of levels of recall and 
precision according to his chosen policy. 

6.2. Hierarchical searching . In the present experiment, there were many 
examples of documents which were chosen as relevant which had been indexed 

15 L. Rolling, o£. cit . 
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by a term UDC number which represents a concept hierarchically related, 
but more specific than that specified by the search formulation. Such 
documents were retrieved by treating the UDC requested number as a *'root*'. 
There were no failures attributable to this technique. It is highly 
recommended that all searches be run in this manner. In terms of the 
system described in this paper, all UDC numbers should appear with a dollar 
sign ($) suffixed to them. 

It has sometimes been suggested that a retrieval system which useT'a^ “ 
hierarchical code such as UDC should be programmed to automatically search 
at one level higher in the hierarchy if no documents are found at the term 
requested. The reader should note that this strategy is contradicted by 
our recommendation that feedback from the system be used in formulating 
searches. According to the policy recommended here, no search would request 
a UDC class number to be present which had not been used as to index at 
least one document, even though the class may exist in theory in the UDC 
schedule. Thus, if a UDC class number is found not to have been used in 
the system, it is up to the search analyst to specify whether or not the 
number for a more general concept should be used. 

6.3. New indexing policies . We have suggested, without prejudice to the 
Iron and Steel Institute, whose use of the UDC was based on policies devised 
for a manual information retrieval system, that certain new policies should 
be considered in order to effect improved performance of a mechanized system. 

The analysis of failures indicated three areas in which indexing policy 
could be modified to the probable benefit of system performance in a computer- 
based system. 1‘iey are (l) specificity of indexing, (2) depth of indexing, 
and (5) consistency of indexing. 







m 







>p 



-50 



6.3«1» Specificity of indexing . When several closely related topics are 
subjects of a document, traditional policy, based originally on single entry 
catalog practices prescribed use of a general class which describes the 
subjects collectively. For example, although specific properties of metals 
were often mentioned in ABTICS abstracts, they were often indexed simply by 
539 . 4 /. 5 , physical properties. When UBC is employed in a computer-based 
system, we recommend that each specific topic be indexed separately. 

6 . 5 . 2 . Depth of indexing . Once again owing to the need to limit the number 
of entries made in a manual card file, traditional policy has been to index 
only the several most important subjects. In the present case, the average 
was observed to be about three UBC class numbers per document. 

In the computer-based system, with more compact storage and fast access 
to multiple index points, there is no need to exercise such stringent control. 
While there is no "magic number" for depth of indexing, the evidence of this 
limited test indicates that more UBC class numbers could be assigned to the 
average document to advantage. A useful rule of thumb mi^t be to make 
index entries for each distinctly identifiable topic which appears in the 
title and abstract of the document. 

6.5.3. Consistency of indexing . In section 5 . 2. 1.2.1, a possible range of 
facets or aspects of metallurgy and its related subjects was outlined. We 
now suggest that improved performance mi^t result from use of an indexing 
work sheet which reminds the indexer to consider each of the possible facets 
which may be anticipated. A sample work sheet is shown in Figure . Of 
course, each information system which uses UBC would need to devise its own 
form on the basis of subject matter and expected user requirements. 
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PIGUEE 12 



A Suggested TJDC Indexing Work Sheet for Metallurgy 



1, Abstract Number • 

2. Place (Country) 

5 . Time (generally for production statistics) 





Unit or Link 1 !' 


Unit or Link 2 


Unit or Link 3 


4* Material ( element > mineral* 
compound, manufactured 
product, etc.) 








5 . Process (initiated by man) 








6. Phenomenon (natural process) 








7 . Machine, apparatus, device 








6 . Complex systems 

(e.g. manufacturing plants, 
industries) 








9 . Purpose, intended use 








10. Physical or chemical state 








11. Measurable property of material 








12. Measurable property of a 
process 








13 . Form, shape of material 








14 . Position, location 
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6.4 Revisions and Innovations in the UDC . We have presented a detailed 
discussion of the difficulties which arise at the time UDC-indexed files 
are searched which are attributable to the structure and class definition 
system of the UDC (see section 5.2.1.). The problems of revisions and 
innovations reflect a deep-rooted question for the International Federation 
for Documentation: can the UDC be universal in the sense of being applicable 
to all types of information systems? Are the requirements of organizations 
which will use the UDC for the purpose of systematic single-entry document 
file organization (e.g. conventional libraries) compatible with those of 
organizations which will offer services based on deep indexing, highly 
specific questions, and the use of the computer as an aid? 

From the point of view of the latter type of system, continued revision 

of UDC according to principles and techniques of faceted classifications 

seems to be indicated. We also recommend the testing of more sophisticated 

devices for coding syntagmatic relationships, such as the schema of relators 

l6 

suggested by Perreault. 

6.5. General Conclusion . The experiment reported here should lend support 
and encouragement to those who will consider use of UDC in computer-based 
retrieval systems. No insoluble problems were found, but the long-existent 
matter of the theory according to which the UDC will be developed in the 
future is seen to be accentuated by the requirements and capabilities of 
computer-based systems. Although the data were limited to results from 
twenty-five search requests run against a file of less than 10,000 document 

16. It is noted with satisfaction that a project which includes this aim 
commenced activity in December, 196?» under the direction of Mr. Thomas 
Caless at the George Washington University in Washington, D.C. 
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rerGrGxicGS) thG pGrfonnanc© charact©ristics deriyed for the system were 

entirely satisfactory. It is to be hoped that further tests on a considerably 

larger scale will be conducted in the near future in order to bridge the 

gap between the laboratory and the real information system. 
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Appendix 1 

Discussion of Methodology 

Observations (or estimates) from each search were made for the 
following document retrieval profile: 





Relevant 


Document Category 
Not Relevant 


Total 


Retrieved 


^11 


^21 


^1. 


Not Retrieved 


^21 


CM 

CM 


^2. 


Total 


^1 


^2 


Y 

• • 



Relevancy of retrieved documents was judged by two persons (J-j» J 2 ) s't 
two levels, relevant and marginally relevant. Documents judged relevant 
at these two levels are designated by a single and double prime (e.g. 
and Y'p. 

Two independent searches were performed using the same questions the 
first search was made by the system under study and the second search was 
made by another system. Documents retrieved by each system were sent to 
the judges for their assessment of relevance, The judges had no knowledge 
concerning which system produced which documents. Results of the search 
and corresponding judgements of relevance are given in Table 1 for documents 
judged relevant and for documents judged to be either relevant or marginally 
relevant respectively. 

Total retrieval (Y. ) is estimated from observed system retrieval only 
from the system under study. Relevant retrieved (Y^^) is estimated from 
judgements made from the subset (Y. ) found by the system under study. The 
recall ratio is based only on relevant documents found by the other system. 
Thus, it is assumed that relevant documents found by the other system is a 
representative sample of all relevant documents in the file. The portion 
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of those documents also retrieved by the system under study is the 
numerator of the recall ratio and the number of relevant documents chosen 
by the other system is the denominator. The fact that each system retrieved 
significant numbers of documents covered by, but not retrieved by the other 
system led us to the conclusion that this technique produces a more accurate 
estimate of the recall ratio than would be obtained by using the ratio of 
relevant retrieved by the system (Y^^) divided by the total relevant found 
by both systems. Total relevant documents (Y .) is estimated by dividing 
(Y^^) by the recall ratio. 

Since the data for estimated recall, estimated relevant documents, 
and precision were available as ratios (proportions), they were analyzed 
by applying an arcsin of the square root of the proportion transformed on 
each data point, calculating the averages and their standard errors, and 
then re-transforming these results into percentage terms. Averages in 
Tables 2 , 3» and 4 (Appendix II) were computed by summing the appropriate 
arcsin transformation values of the questions, dividing by the total number 
of questions, and re- transforming the results. The averages for total 
retrieval were calculated dir^^ctly without a transformation. 

The calculation of the standard errors around each of the averages is 
more cumbersome. To compute these standard errors, the variance from the 
sample must be estimated. The standard error around each particular 
variable is found by dividing the variance by the number of observations 
for a specific variable (e.g. Judge l), and taking the square root of that 
number. The most direct method of obtaining the variances is to calculate 
the total and regression (explained) sum of squares. The difference between 
them gives the residual (= error = unexplained) sum of squares, which when 
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divided by the corresponding number of degrees of freedom yields the 
variance (mean square) . The square root of this variance is the standard 
deviation. The data resulting from the above computations is presented 
in analysis of variance tables for each of the samples. 
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Formulas used in computations 



Let -n = total number of observations 

n^^ = number of observations for Judge 1 



Y = total number of degrees of freedom 

Vg = explained degrees of freedom; = r 
freedom 

X. .= observation, i = 1, 2; j = 1, n 

^3 

A. Observations and Degrees of Freedom 
n = Hi + 



where No. of Blocks = 2 = Judge 1, Judge 2 = i 



n^ = number of observations for Judge 2 



n - 1 



y = (No. of Blocks) -1 

Tt* 





B. Sums of Squares 



Total 




# 



i = 1, 2 = Judges 



n 



j = 1, ...n = no. of questions 
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Explained 



ESS 



. \,) ^ . (I\ ) ^ (I?..) 



n. 



Residual 



n. 



n 



RSS = TSS - ESS 



C. Variance 



= variance (total) ^ 

2 

C7_ = variance (unexplained) 

R 

For Average (per Judge) 



2 RSS 

— = Residual Mean Square 
R 



For Total Average 

^ 2 __ TSS 
T "* y 

D. Standard Error (SE) 



For Judge 1 



SE 



- + \l- - ± 4 ^^^ 

Yn^ YVi 



For Judge 2 



SE 



For Total 



SE 






-± j 



TSS 

yn 
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E. Averages 



Judge 1 (J^) 



2^ X 

7 , j-1..^ 

‘^1- n. 



Judge 2 (J_) 



X . 

2 nj 



Total (T) 



T = 



n 

£ X 

i=l ii 
n 



Li w 













n yr^^7W-i r^ ij-P^W|»j;’^n.^Jt!.wwy.^i!>.»i null I Ut'l lUJlBlUiWBffBggjg; 
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Appendix II 

Computation of Performance Measures 
Table 1: Number of References Retrieved and Judged Relevant 



■ 

Question 


Number 

Retrieved 


Number Judged 
Relevant 

Judffe 1 Judffe 2 


Number Judged I 
Relevant or Marginal I 
Judffe 1 Judge 2 1 






Y' 

^11 


y" 1 

11 


1 


8 


^ ! 


7 


8 


7 


2 


18 


7 


16 


16 


18 


5 


18 


1 


6 


16 


18 


4 


25 


2 


15 


4 


22 


5 


- 


- 


- 


- 


- 


6 


5 


5 


5 


5 


5 


7 


16 


7 


2 


11 


2 


6 


8 


0 


6 


0 


6 


9 


2 


2 


1 


2 


2 


10 


- 


- 


- 


- 


- 


11 


2 


2 


2 


2 


2 


12 


2 


2 


2 


2 


2 


15 


- 


- 


- 


- 


- 


14 


- 


- 


- 


- 


- 


15 


45 


19 


26 


26 


29 


16 


- 


- 


- 


- 


- 


17 


1 


1 


1 


1 


1 


18 


218 


16 


19 


54 


85 


19 


40 


0 


0 


1 


5 


20 


2 


2 


2 


2 


2 


21 


16 


5 


16 


15 


- 


22 


62 


16 


9 


5 


60 


25 


25 


10 


8 


19 


19 


24 


185 


141 


158 


177 


180 


25 


18 


10 


17 


10 


17 
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Table 2 : RECALL RATIOS 



I 



(Question 


Relevant 


Combined 




Judge 1 


Judge 2 


Judge 1 


Judge 2 


1 


0.333 


0.250 


0.200 


0.125 


2 


0.200 


0.500 


0.333 


0.300 


4 


1.000 


0.500 


1.000 


0.333 


6 


0.500 


0.400 


0.500 


0.400 


7 


0.500 


1.000 


0.333 


0.333 


8 


0.000 


0.200 


0.000 


0.200 


15 


0.000 


0.167 


0.500 


0.125 


18 


0.000 


. 1.000 


0.125 


0.200 


21 


0.667 


0.714 


0.714 


0.625 


22 


0.444 


1.000 


0.500 


0.600 


23 


0.125 


0.250 


0.077 


0.143 


24 


0.333 


• 1 .000 


0.333 


0.333 


Average 


0.295 


0 . v 668 


0.373 


0.299 


i SE 










(per Judge) 


J; 0.018 


i 0.018 


i 0.008 


± 0.008 


Total Average 


O.48O 


0.336 


± SE 


±0 


.011 


±0.004 



Analysis of Variance 
RECALL RATIO : RELEVANT 



Source of 
Variation 


Degrees of 
Preedom 


Sum of 
Squares 


Mean 

Square 

(Variance estimate) 


Total 


23 


I8OI3.5OI 


— 


Judges 


1 


2888.621 


2888.621 


Residual 


22 


15924. 880 


723.858 



_ 2888.621 _ , gg 

^ " 723.858 “ 

P {^0 level of variance ratio) = 4*30 
P ( 1 ^ level of variance ratio) = 7«94 






; 






I 

1 

f 



RECALL RATIO : COMBINED 



Source of 


Degrees of 


Sum of 


Mean 


Variation 


Preedom 


Squares 


Square 


Total 


23 


6974.717 


— 


Judges 


1 


120.826 


120.826 


Residual 


22 


6853.891 


311.541 
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Table 5t ESTIMTED TOTAL RELEVAMT 



Question Relevant 



Combined 



1 

2 

4 

6 

7 

8 
15 
10 
21 
22 

23 

24 


Judge 1 

24.00 

35.00 
2.00 
6.00 

14.00 
0.00 
0.00 
0.00 
7.50 

36.00 

152.00 

423.00 


Judge 2 

28.00 

32.00 

30.00 
7.50 
2.00 

30.00 
159*00 

19.00 
22.40 

9.00 

32.00 
158.00 


Judge 1 

40.00 

48.00 

4.00 

6.00 

33.00 
0.00 

52.00 

432.00 
10.20 

62.00 

247.00 

531 .00 


Judge 2 

56.00 
26.67 

66.00 
7.50 
6.00 

30.00 

232.00 

425.00 

100.00 

133.00 

540.00 


Average Rel. 


58.29 


44.00 


122.77 


147.47 


± SE 










(per Judge) 


±26.99 


+26 . 99 


+52.11 


154.43 


Tottl Average 


51. 


.10 


134.58 


Relevant 










+ SE 


+19.39 


+37.28 






i 

% 



Analysis of Variance 
EST. TOTAL RELEVANT : RELEVANT 



Source of 


Degrees of 


Sum of 


Mean 


Variation 


Freedom 


Squares 


Square 


Total 


23 


190429.65 


— 


Judges 


1 


6161.05 


6161.05 


Residual 


22 


192267.00 


8739.45 



EST. TOTAL RELEVANT : COMBINED 



Source of 


Degrees of 


Sum of 


Mean 


Variation 


Freedom 


Squares 


Square 


Total 


22 


607866.37 


mtam 


Judges 


1 


3503.10 


3503.10 


Residual 


21 


604363.27 


32588. 73 
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Table 4 : PRECISION RATIOS 

Question Relevant Combined 





Judge 1 


Judge 2 


Judge 1 


Judge 2 


1 


1.000 


0.875 


1.000 


0.875 


2 


0.389 


0.889 


0.889 


1.000 


3 


0.056 


0.333 


0.889 


1.000 


4 


0.087 


0.652 


0.174 


0.957 


6 


1.000 


1.000 


1.000 


1.000 


7 


0.438 


0.125 


0.688 


0.125 


8 


0.000 


0.750 


0.000 


0.750 


9 


1.000 


0.500 


1.000 


1.000 


11 


1.000 


1.000 


1.000 


1.000 


12 


1.000 


1.000 


1.000 


1.000 


15 


0.442 


0.605 


0.605 


0,674 


17 


1.000 


1.000 


1.000 


1.000 


18 


0.074 


0.087 


0.248 


0.390 


19 


0.000 


0.000 


0.025 


0.075 


20 


1.000 


1.000 


1.000 


1.000 


21 


0.313 


1.000 


0 . 8.3 




22 


0.258 


0.145 


0.500 


0.968 


23 


0.760 


0.320 


0.760 


0.760 


24 


0.770 


0.853 


0.967 


0.984 


25 


0.556 


0.944 


0.556 


0.944 


Average 


0.615 


0.723 


0.780 


0.899 


+ SE 










(per Judge) 


io .015 


iO .015 


iO.011 


±0.011 


Total Average 


0.666 


0.842 


+^SE 


jO .007 


+0.006 



mmmmm 

I 

5 









ttiaiiiiitffliiil 



m 



itiiiar*ihtkrtiateto.tariairitaicrt^Wr.ia^^^ 















- 64 - 

Table 4 Cont. 



Analysis of Variance 
PRECISION RATIO : RELEVANT 



Source of 
Variation 


Degrees of 
Freedom 


Sum of I 

Squares 


Mean 

Square 


Total 


59 


37175.846 


— 


Judges 


1 


450.750 


450.750 


Residual 


38 


56745.116 


966.977 



PRECISION RATIO : COMBIIHD 



Source of 
Variation 


Degrees of 
Freedom 


Sum of 
Squares 


Mean 

Square 


Total 


38 


26789.125 


— 


Judges 


1 


866.500 


866.500 


Residual 


37 


25922.625 


700.611 
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Continuation of Section 4 - Experimental data 
from pages 24 and 25 
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4. 



1, Question 




mmmmmm 



. 1 
• 1 
. 1 



. 1. Question 

• 1«1« Original Statement Blsist/ PurnscGS &nd Tuy©I*©S 

,1.2. Added Notes Any association of Tuyeres with blast 
furnacGs, blast furnace practice or Iron-making. 



. ][ .1.3. UDC Analyst's Notes 

Blast Furnaces - 669.162.2. In the most general sense, of 
description and broad view of operations. 



Blast Furnace Tuyeres - 
specifically "tuyeres”, 
is 669*162.221.8. 



669.162.221.2. This is quite 
(normal, hearth tuyeres) Bosh Tuyeres 



i 






i 



4. 1 .2. UDC Descriptors Chosen 

Line Descriptor Encoded Descriptor Frequency of Posting 
A 669.162.221.2 C669I6222I2 8 

B 669.162.221.8 C669I6222I8 0 

(Note: All searches were run without regard to frequency of 
posting. A real information system would take this factor 

into account.) 



. 1 . 3. Encoded Logical Statements (condensed form) 
(+ ss Of j * as and) $ ~ truncation) 

1. (A or B) 

2. (A$ or B$) 
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4. 1.4. Results and Analysis 
4. 1.4.1. Relevance Judgements 





Character- 

istic 


Judges 


Total 


Relevant 


Marginally 

Relevant 


Non- 

Relevant 






Retrieved 
by Both 
Systems 


Judge 1 
Judge 2 


1 

1 


1 

1 


0 

0 


0 

0 


Retrieved 
by UDC 
only 


Judge 1 
Judge 2 


7 

7 


7 

6 


0 

0 


0 

1 


Total 
Retrieved 
by UDC 


Judge 1 
Judge 2 


8 

8 


8 

7 


0 

0 


0 

1 


Retrieved 
by ASM 
only^ but 
covered by 
-ISI ABTICS 


Judge 1 
Judge 2 


7 

7 


2 

3 


2 

4 


3 

0 


Total Re- 
trieved by 
both sys- 

terns (Line 
1) and by 
ASM only 


Judge 1 
Judge 2 


8 

8 


3 

4 


2 

4 


3 

0 





3 



4. 1.4.2. Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


Estimated No. of 


Precision 


Estimated 

Specificity 


Base 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

•T 

Marginal 


Relevant 

only 


Relevant 

+ 

Marginal 


Formula 


B 


B+C 


J 


J +K 


J 


J + K 


JR 


(J-HO (R-»S 


R 


R+S 


B/R 


(B-KJ)/(R^S 


I 


I 


9159 B 


9159(B^-c: 


Judge 1 


1 

3 


1 

5 


8 

“17I 


8 

175 


oc|oo 


oc|oo 

1 


• 0024 


• 0044 


Judge 2 


1 

IT 


1 


7 


7 

I7ff 


7 


7 


.0031 

■ 


• 0061 
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4* <• Question 2 



4. 2'.1. Question 

4. 2 .1.1. Original Statement Retained Austenite 



4. 2 .1,2. Added Notes Any reference or mention of retained 

austenite. 



4. 2 .1.3. UDC Analyst's Notes 

Retained Austenite - 669.112.227.343. This Is quite specific. 
Transformation of retained austenite Is 669.112.227.346.3. 



4. 2 .2. UDC Descriptors Chosen 




Line Descriptor Encoded Descriptor 


Frequency of Posting 


A 669. 112. 227. C669112227343 


12 


B 669.112.227.3^6.3 C6691122273'l63 


6 



4. 2 . 3. Encoded Logical Statements (condensed form) 



(+ = or, * * and, $ = truncation) 



1 . 

2 . 



(A or B) 
(A$ or B$) 
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A. 2 *A. Results and Analysis 
A. 2 .A.l. Relevance Judgements 
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A. .A. 2. Derived Performance Characteristics 
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4.3 Question 3 



4. 3 .1. Question 

4. 3 .1.1. Original Statement Alloying Effects on Strain Aging and 

Internal Friction of Carbon Steels 
4. 3 .1.2. Added Notes Effect of Cr on strain aging behavior of low 
carbon steel. Internal friction of low carbon steels. Influence 
of Mn and Cu. on internal friction of low carbon steels. Strain 
aging of low carbon steels. Internal friction**Snoek mechanism 
elasticity. 

4. 3 .1.3. ODC Analyst's Notes 

This is best as a broad search under 539.67 for internal friction, 
and 621.785.797 for strain aging; connection by colon (:) with 
669.26 for Cr, 669.3 for Cu, and 669.74 for Mn will narrow it. 



L 3 .2. 


UDC Descriptors Chosen 




Line 

A 


Descriptor 

539.67 


Encoded Descriptor 
C53967 


Freouency of Postlne 
27 


B. 


621.785.797 


C621785797 


7 


C 


669.26 


C66926 


28 


D 


669.3 


C6693 


30 


E 


669.74 


C66974 


40 



4. 3.3. Encoded Logical Statements (condensed form) 
(+ ■ or, * ■ and, $ ■ truncation) 

1. C(A ♦ B) * (C ♦ D ♦ E)1 

2. (A$ B$) 
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4,3 .A, Results and Analysis 



4.3 .4.I. Relevance Judgements 





Character- 

istic 


Judges 


Total 


Relevant 


Marginally 

Relevant 


Mon- 

Relevant 




Retrieved 
by Both 
Systems 


Judge 1 
Judge 2 


0 

0 


0 
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0 
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Retrieved 
by UDC 
only 


Judge 1 
Judge 2 


18 
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12 
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0 


Total 
Retrieved 
by UDC 


Judge 1 
Judge 2 


18 
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15 

12 


2 

0 


Retrieved 
by ASM 
only^ but 
covered by 
ISl ABTICS 


Judge 1 
Judge 2 


4 
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1 

0 


3 

3 


Total Re- 
trieved by 
both sys- 

terns (Line 
1) and by 
ASM only 


Judge 1 
Judge 2 


4 

4 


1 


1 

0 


3 

3 



4. 3.4.2. Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


Estimated Mo. of 


Precision 


Estimated 

Specificity 


Base 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

’+ 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

+ 

Marginal 


Formula 


B 


B**“C 


J 


J +K 


J 


J-»“K 


JR 


rj-40 (R-»S 


R 


R+S 


B/R 


(B*K:)/(R+S 


I 


1 


9159 B 


9159(B»-C; 


Judge 1 


n.a. 


n.a. 


n.a. 


11 • 81 • 


1 

TT 


16 

TF 


n • 81 • 


n • 81 • 


Judge 2 


n • 81 • 


n.a. 


n • 81 • 


n • 81 • 


6 


18 

TF 


n • 81 • 


n.a. 

• 










44 Question 4 



4. 4.I. Question 

4« Original Statement Joining of Thin SS or Ni -Alloy Sheets » 

^ strip or foil 

4. 4 .1.2. Added Notes Fusion welding or resistance spot welding of 
thin stainless steel or nickel base alloy sheet » or strip » or foili 



4. 4 .1.3. UDC Analyst’s Notes Electric fusion welding is 621.791.753» 
submerged arc welding 621.791.753.5, and welding with protective 
atmospheres is 621.791.753.9: resistance spot-welding is 
621.791.7153.1. Stainless steels are 669.14.018.8, where the 
composition is not specified. When it is, as in a Cr-Ni^ stainless > 
it would have the number 669.15*24*26-194.56, to denote an 
austenitic (-194.56) alloy steel containing Cir or Ni: other 
elements, such as Mo, would cause the insertion of *28 af ter the 
(conx. in borrom box) 



4. 4 


.2. UDC Descriptors Chosen 




Line 


Descriptor 


Encoded Descriptor 


Freouency of Posting 


A 


621.791.753 


C621791753 


75 


B 


621.791.763.1 


C6217917631 


4 


C 


669.14 


C66914 


2073 


D 


.018.8 


X188 


393 


E 


669.15*24 


C66915Y24 


139 


F 


669.245 


C669245 


3 


G 


•415 


W415 


101 


H 


•416 


W416 


11 


I 


•418 


W418 


18 



1 

l 



i 



4. 4 . 3. Encoded Logical Statements (condensed form) 

(+ ■ or, * * and, $ ■ truncation) 

1. ♦ B) • (j(C*D$) ♦ n ♦ F] * (G$ ♦ 11$ ♦ 1$^ 

2. Am * B$) * 0(C$ • D$) ♦ E$ ♦ P$] • (G$ ♦ li$ ♦ 1$^ 

3. /621.791.7I * CfCS * D$1 * E$ ♦ PS]")- 

Ter nickel alloys, unspecified, are -669.245, and when the com- 

■posltton Is given, e.g. a Nl-Cr alloy, -denoted by 669.255'2o. The 
bhape is denoted by -415, for thin plate and sheet, -416 for very 
thin plate and foil, and -4l8 for strips. Various combinations 
must therefore be made, as for example: 
669.14.018.8-416:621.791.763.1 Spot welding of stainless steel 

foil. 
















itt 



liMllllglliiiiHii 



4.4 .4. Results and Analysis 



4.4 .4.1. Relevance Judgements 



Character- 

istic 


Judges 


Total 


Relevant 


Marginally 

Relevant 


Non- 
Re levant 


Retrieved 
by Both 
Systems 


Judge 1 
Judge 2 


1 

1 


1 

1 


0 

0 


0 

0 


Retrieved 
by UDC 
only 


Judge 1 
Judge 2 


22 

22 


1 

14 


2 

7 


19 

1 


Total 
Retrieved 
by UDC 


Judge 1 
Judge 2 


23 

23 


2 

15 


2 

7 


19 

t 

1 


Retrieved 
by ASM 
only^ but 
covered by 
-ISI ABTICS 


Judge 1 
Judge 2 


2 

2 


0 

1 


0 

1 


2 

0 


Total Re- 
trieved by 
both sys- 

terns (Line 
1) and by 
ASM only 


Judge 1 
Judge 2 


3 

3 


1 

2 


0 

1 


2 

0 



4. 4*4.2. Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


Estimated No. of 


Precision 


Estimated 

Specificity 


Base 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

+ 

Marginal 


Formula 


B 


B+C 


J 


J +K 


J 


J-»-K 


JR 


t J-HCl (R^ 


R 


R+S 


B/R 


(b-k;)/(r-»s 


I 


I 


9159 B 


9159(B^c: 


Judge 1 


1 


1 


2 


4 


2 


4 








1 


1 


ITT 


ITT 


23 


23 


• 0002 


• 0004 


Judge 2 


1 


1 


15 


22 


15 


22 








2 


3 


1/2 


1/3 


23 


23 


• 0033 


• 0072 







i 
















II I I III 




















mmmmmm. 



mrnrnmmmmmmmmmm 







4.5 



Question 




4.5 « 1 . Question 

4.5 .1.1. Original Statement Effect of Alloying Additions on the Austenitic 

Properties of Steels 

4. 5 .1^2 Added Notes Effect of small additions of aluminum, niobium 
(columbium), vanadium, titanium, and zirconium on the austenitic grain size 
grain coarsening temperature, mechanical properties, forgeability and 
impact properties of plain carbon bar, rod, wire and tube steels in the 
presence of varying amounts of nitrogen and sulfur. Solution temperature 
of the oxides, carbides, nitrides, and sulfides of the above elements in 
austenite. Solubility product of aluminum, columium, chromium, vanadium, 
titanium, and zirconium oxides, carbides, nitrides, and sulfides in 
austenite as a function of temperature. 

4. 5 .1.3. UDC Analyst's Notes 

This is a severe test for any scheme of classification or any thesaurus or 
controlled vocabulary. However, as the "definition" refers, largely, to 
austenite 669.112.227.1 provides a good general handle. Abnormal grain 
size will be denoted by -175, and the study of grain size, as a metallo- 
graphic technique, by 620.186.82; mechanical properties should be sought 
either as a general subject at 539.4/. 5, or under individual properties. 
Forgeability is best sought under 621.733.01. The aspect of "bar, rod, 
wire or tube" are hyphen numbers and according to my practice, would be 
found at 669.14-422.1, 669.14-422.2, 669.14-426, and 669.14-462 

solution and solubility aspects should be found at 
541.8, solution and solubility in general. The various elements affecting 
of the properties will have been classified under their numbers from 
® oxides, carbides, etc., by means of the numbers from 546, e.g. 

546e621*171el, aluminium nitride^ 669«296 zirconium. 
546.831*261 zirconium carbide. 








mtm 













4. 5 . 


2. UDC Descriptors 


Chosen 




Line 


Descriptor 


Encoded Descriptor 


Frequency of Posting 


A 


669.71... 


C66971$ 


61 


B 


669.29. . . 


C66929$ 


91 


C 


669. . . 


C669$ 


very many 


D 


620.186.32. . . 


062018682$ 


10 


E 


539.4. . . 


C5394$ 


848 


F 


539.5. . . 


C5395$ 


286 


G 


620.17... 


062017$ 


369 


H 


669.14... 


C66914$ 


very many 


I 


621.733... 


0621733$ 


24 


J 


620.178.7... 


06201787$ 


55 


K 


541.8... 


C5418$ 


19 


L 


669.112.227.1. 


.. C6691122271$ 


46 


M 


546.621 


CS46621 


1 


N 


546.621'171.1 


C546621Y1711 


0 


0 


546.621*261 


C546621Y261 


0 


P 


546.621*221 


C546621Y221 


0 


Q 


546.882 


0546882 


0 


R 


546.882*171.1 


C546882Y1711 


0 


S 


546.882*261 


C546882Y261 


2 


T 


546.882*221 


C546882Y221 


1 


U 


546.881 


0546881 


0 


V 


546.881*171.1 


C546881Y1711 


0 


w 


546.881*261 


C546881Y261 


3 


X 


546.881*221 


C546881Y221 


0 


Y 


546.821 


0546821 


1 


Z 


546.821*171.1 


C546821Y1711 


0 


AA 


546.821*261 


C546821Y261 


2 


AB 


546.821*221 


C546821Y221 


1 


AC 


546.831 


0546831 


0 


AD 


546.831*171.1 


C546831Y1711 


0 


AE 


546.831*261 


C546831Y261 


0 


AF 


546.831*221 


C546831Y221 


0 


AG 


-175... 


W175$ 


4 


All 


-31... 


W31$ 


48 


AI 


-422. . . 


W422$ 


24 


AJ 


-426. . . 


W426$ 


57 


AK 


-462... 


W462$ 


117 


AL 


.01. . . 


Xl$ 


very many 


AM 


.018.2... 


X182$ 


487 



^•5 


.3. Encoded Logical Statements (condensed form) 
(+ = or, * * and, $ ® truncation) 




1. 


■^A + B) * t (C • AG) + D + E + F*G + (H* AM) ♦ (I 


• AL) ♦ JJ * 




tC • (AI ♦ AJ + AK)^ 




2. 


^ • L • {CAH »(M + Q+ U + Y + AC)J + N ♦ 0 + P + 


R+S + T + V+W 




+ X ♦ 2 ♦ AA ♦ Al^ 












I 4, 5.4, Results and Analysis 



4. 5.4.1. Relevance Judgements 





Character- 

istic 


Judges 


Total 


Relevant 


Marginally 

Relevant 


Non- 

Relevant 




Retrieved 
by Both 
Systems 


Judge 1 
Judge 2 


0 

0 


0 

0 


0 

0 


0* 

0 


Retrieved 
by UDC 
only 


Judge 1 
Judge 2 


0 

0 


0 

0 


0 

0 


0 

0 


Total 
Retrieved 
by UDC 


Judge 1 
Judge 2 


0 

0 


0 

0 


0 

0 


0 

0 


Retrieved 
by ASM 
only, but 
covered by 
-ISI ABTICS 


Judge 1 
Judge 2 


6 

6 


4 

4 


1 

1 


1 

1 


Total Re- 
trieved by 
both sys- 

terns (Line 
1) and by 
ASM only 


Judge 1 
Judge 2 


6 

Q 

6 


4 

4 


1 

1 


1 

1 



4. 5 .4.2. Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


Estimated No. of 


Precision 


Estimated 

Specificity 


Base 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

+ 

Marginal 


Formula 


B 


B + C 


J 


J +K 


J 


J + K 


JR 


( J-HO (R^S' 


R 


R+S 


B/R 


(B^C)/(R-^S 


I 


1 


9159 B 


9159(B<-c: 


Judge 1 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 


Judge 2 


n.a. 


n«a. 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 
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A. 6 Question 6 



A* 6 .1. Question 

A. 6 .1.1. Original Statement Tubes— Cold Drawing 



A. 6 .1.2. Added Notes 



Cold drawing or cold rolling of tubes of any type 
of steel. 



A. 6 .1.3. UDC Analyst's Notes The definition refers to cold rolling or cold 
drawing. The former Is 621.774.35.016.3, the latter Is 621.774.37.016.3. 



A. 6 


.2. UDC Descriptors 


Chosen 




Line 


Descriptor 


Encoded Descriptor 


Frequency of Posting 


A 


621. 77*1. 35 


C62177435 


28 


B 


621.774.37 


C62177437 


5 


C 


621.774.3$ 


C6217743S 


60 


D 


• 0X6 • 3 • • • 


X163$ 


90 



A. ^ .3. Encoded Logical Statements (condensed form) 



(+ = or, * = and, $ *= truncation) 
1. C(A + B) » D] 




ifeaiaaaaa8iiBaiiii^iateagaaa6Biriaiiii^^ 









4. 6«4. Results and Analysis 
4. 6.4.I. Relevance Judgements 



Character- 
istic 



Judges 



Total 



Relevant 



Marginally 

Relevant 



Non- 

Relevant 



Retrieved 
by Both 
Systems 



Judge 1 
Judge 2 



2 



B 



0 

0 



0 



0 



Retrieved 
by UDC 
only 



Judge 1 

«• w «• 

Judge 2 



0 

0 



0 



H 



0 



Total 
Retrieved 
by UDC 



Judge 1 



J - 



Judge 2 



K 



0 

0 



0 



0 



Retrieved 
by ASM 
only, but 
covered by 
‘ISl ABTICS 



Judge 1 
Judge 2 



4 



M 



N 



4 



0 



Total Re- 
trieved by 
both sys- 
tems (Line 
1) and by 
ASM only 



Judge 1 
Judge 2 



4 



4 



0 

1 



4.6 .4.2. Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


Estimated Mo. of 


Precision 


Estimated 

Specificity 


Base 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

•+ 

Marginal 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

+ 

Marginal 


Formula 


B 


B + C 


J 


J +K 


J 


J+K 


JR 


(J-40 (R-»«: 


R 


R +S 


B/R 


(B-K3)/(R4S 


I 


I 


9159 B 


9159(Bt-C 


Judge 1 


2 

T 


2 

T 


3 

ITT 


3 

ITT 


3 

7 


3 

7 


.0007 


.0007 


Judge 2 


2 


2 


3 


3 


3 


3 








4 


5 


274 


275 


3 


3 


.0007 

' 


.0008 




. 









4.7 Question 






4.7 .!• Question 

4*^ .l.l. Original Statement Plug Mill Tube Making 

4.7 .1.2. Added Notes tolling of Steel seamless tubes in plug rolling mill 
and rolling conditions including torque, stresses, strains, roll forces, 
and forces acting upon mandrel. 

4.7 .1.3. UDC Analyst's Notes 

I take this to be rolling over a plug set in the roll gap, for the further 
expansion of already pierced tubes. This will therefore be denoted by 
621.774.35: the auxiliary aspects, such as torque, roll-force, etc., will 
be indicated by adding subdivisions of .01, e.g. .011, stress and flow of 
material, .014.3, roll pressure. 



4.7 .2. UDC Descriptors Chosen 




Line Descriptor 


Encoded Descriptor 


Freouency of Post Ins 


A 621.774.35 


C62177435 


28 


B . 01 . . . 


Xl$ 


very many 


C 621.774.3... 


C6217743S 


68 



4.7 .3. Encoded Logical Statements (condensed form) 
(+ " or, * ** and, $ ■ truncation) 

1. (A * B) 

I 2. (A$ • B) 

3 . (C • B) 














4*7 .4. Results and Analysis 



4.7 .4.1. Relevance Judgements 



Character- 

istic 


Judges 


Total 


Relevant 


Marginally 

Relevant 


Non- 

Relevant 


Retrieved 
by Both 
Systems 


Judge 1 
Judge 2 


1 

1 


1 

1 


0 

0 


0 

0 


Retrieved 
by UDC 
only 


Judge 1 
Judge 2 


15 

15 


6 

1 


4 

0 


5 

14 


Total 
Retrieved 
by UDC 


Judge 1 
Judge 2 


16 

16 


7 

2 


4 

0 


5 

14 


Retrieved 
by ASM 
only, but 
covered by 
-ISI ABTICS 


Judge 1 
Judge 2 


4 

4 


1 

0 


1 

2 


2 

2 


Total Re- 
trieved by 
both sys- 

terns (Line 
1) and by 
ASM only 


Judge 1 
Judge 2 


5 

Q 

5 


2 

1 


1 

2 


2 

2 



i 



\ 



4. 7.4.2. Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


Estimated No. of 


Precision 


Estimated 

Specificity 


Base 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


I Relevant 
Marginal 




B 


B + C 


J 


J +K 


J 


J+K 


JR 


(J-40(R4S 


Forinuld 


R 


R+S 


B/R 


(B-»C)/(R-»S 


I 


I 


9159 B 


9159(B*-C: 


Judge 1 


1 


1 


7 


11 


7 


11 








7 


7 


T77 


T77 


TF 


IF 


.0015 


.0036 


Judge 2 


1 

T 


1 

3 


2 

T7T 


2 

m 


2 

IF 


2 

TF 


.0002 


.0007 








mimM 















CO 



•o 

% 

01 



01 

U 



•o 

% 

01 



o 

a 



ca 



01 

u 



c 

01 



CO 




CM 



01 



0 




• 

3 




SB 


8 


O 

1-4 


• ^4 

P 0 




4J 


P • 


• 


• 




pO ^ 


•H f-4 


•o 


p d 


• 


P 


P 


JO 5 


A • 


• S4 




3 ^ 


CO • 


d p 



CO 

CO 

01 

u 

o 

u 

a 



.01 

•5 



CM O 



•2 i 



00 

c 

o 

u 



I 

41 Ql 

CO O 00 
CO O 73 
3 9 

CO 

•O 5 
01 3 CO 



CM Os M 



01 

Os U 
CO O 



CO 

09 

01 

p 

o 

M 

a 



do* 


• • 4J (0 


3 


u p 


M-4 


P 


•H -H • 


m o 




P «H 




o u 


4J • 


• 9 0 


P 


ai-4 


d 


d p 


4J p in 


(0 M 4J 


pC 


P 1-4 


o 




O U CO 


• h 4J 


4J 




p 


P H 


d ^ • 


O CO (0 




>s 9 


p 


•H 




4J 0 h 


4J 


S4 p 


p 




p -H 


« U O « 


P 


P 


u 


P • 


pun- 


« a u M-i 


pC 


•H 1-4 




S4 P 


3 flj • 


CO «M <U 


4J 


•-4 O 


4J 


P iH 


d« ^ 






•H • 


d 


id iH 


1 P CM 


* 00 0 


T3 




p 


H 1-4 


so 


W 0 a 43 


P 


9 P 


u 


B 


p X 


fl» .H o 


4J 


P pC 


p 




P U M-4 


> U 0 «H 


P 


4J 


a 


• 00 


P S4 O 


« 0 OOX 


U 


O 


a 


E d 


U P 


i 0} 14 ^ 


•H 


• M-4 


p 


U -H 


o p d 


O 9 CO 


•o 


•H 




O ^ 


S4 P O 


X « h 


d 


P 


o 


M-4 1-4 


CL -H 


«W >o 0 


•H 


jb ^ 


d 


o 


P 4J 


0 a 




H U 




4J U 


P pC P 


• «H 0 


P 


P 


p 


d 


J3 H U 


4J 1-1 a 


00 


^ -H 


p u 


4J -H 


d "O *H 


•o 


• P 




E P 


M-4 


Q{ 0 B 0 


9 




p 


P pC 


op » iM 






o o 


u 


00 LI 


IS 4J U 


3 X 




•H X 


p 


*o o 


•HUP 


u 0 m 0 




4J 


x: 


9 


u p a 


O 0 w >o 


• 


p 


4J 


s 


d u M 


•o ft 0 


P 


d • 




5 


<U 4J 


dt'O «H 


P 


1-4 m 


d 


P u 


9 (0 


« i4 0 • 


Q 


•o CO 


p 


•H M-4 




•H 0 0 O 0 


O 


u • 


pC 


x: 


m- 4 P 4.1 pC a 4J Li 


1-4 


p 


4J 


4J 



00 

§ 

u 



•o 



o 

d 



p 

00 

•o 

9 



• 

Q> 

P 

d 

5 

p 

• ^ 

p p 

p u 

p I 
p d 
Q 
p d 




8 



O 

P 



p 

S 



P fM 
P CM 



vO 



P 



4i pC CM 

d P M O 

o d 

•M d 4i 

fl 

p 



p 

CM O 



•c 

p 

M 

P 

P 

P 



P 
CM JZ 



P P 
P *1-1 
P *o 
P 



o 4J m 



d p 
P 00 
O P *M «0 
M P P 9 
Ot *M 



P 

JS 



g 



P M 
•M p 

e M 

• p o 

: s w 

p 

d .o 
cri p B 

• 9 

Os d 



CM 



p • 

u -o 

p p 

> > 



p 

8 



p 

P P o 

P 1-4 41 

d 

0) JS 
B P P 
P P ,c 
CO H P 
p 



p • 

p p 
p p 
d 

•o > 

p p 
U IM 
•M P 
9 U 

cr I 

p d 

u o 
d 

Os 



00<M P 
d •M -o 
P B 

I 

- a„ 

•o ^ P 
P IM 5 

M O 
O M P 
d J3 
00 M 
P 9 
OOiM 4i 
*o P P 
® 

•r-l P 4-1 



U 

o 



"S 

p 



d 

o • 

p ,o 
8 CO 

u • 

CM 



P 



• • 4J 

P P P 



^ o 

p 

9 4J 
M d 
P P 

p e 



P Os P 
^ U 
H -H p 
P P 

p a 

• P P 
-O P p 



d in 
p • 
U 

P • 

a<f 

p 

p p 
p 

o 

S5 p 

P 

«5 ^ 



p ^ 
• • ^ 
CO CO 

• • • 

CM CM CM 

• • • 
m m m 

• • • 

• • • 

<f <f <f 



• • 

• ^ ,o 

•M CO CO 

• • • 

CM CM CM 

• • • 

m m m 
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A. 8 Question 8 



A. 8 Question 

A. 8 .1.1. Original Statement Hydrogen Embrittlement of Martensitic and/or 
Austensitic Stainless Steels, and/or Maraging Steels. 

A. 8 .1.2. Added Notes Embrittlement (from any cause, including hydrogen), 
and associated or occurring in any austenitic, martensitic or ferritic 
stainless steel, or in maraging steel. 

A. 8 .1.3. UDC Analyst's Notes 

Embrittlement is denoted by S39.S6; if specifically to hydrogen, 669.788 
will be attached by colon (:). Austenitic, martensitic and ferritic 
stainless steels require -194.56, -194. SS, or -194.57 respectively to be 
added to composition e.g. 669.15*26-194.55, martensitic chromium steels. 



i 



A. 8 


.2. UDC Descriptors Chosen 




line 


Descriptor 


Encoded Descriptor 


Frequency of Post ins 


A 


539. S6 


C53956 


167 


B 


669.788 


C669788 


113 


C 


669.15... 


C66915$ 


very many 


D 


-194.55... 


W19455$ 


21 


E 


-194.56... 


W19456$ 


155 


F 


-194.57... 


W19457$ 


27 



A. 8 .3. Encoded Logical Statements (condensed form) 
(+ or, * » and, $ " truncation) 

1. CA * B ** C ** (D ♦ E ♦ F)] 

2. CA$ ** B$ ** C ** (D ♦ E ♦ F)] 

3. CA$ ** C ** (D ♦ E ♦ F)] 


























4. 8.4. Results and Analysis 
4. 8.4.1. Relevance Judgements 



1 J II U PH -41- 
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Total 


Relevant 


Marginally 

Relevant 


Non- 1 

Relevant 1 


Retrieved 
by Both 
Systems 


Judge 1 
Judge 2 
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0 

1 


0 
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0 1 


Retrieved 
by UDC 
only 


Judge 1 
Judge 2 
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5 
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0 


7 

2 1 


Total 
Retrieved 
by DDC 


Judge 1 
Judge 2 
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0 

0 


8 1 
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Retrieved 
by ASM 
only^ but 
covered by 
-ISI ABTICS 


Judge 1 
Judge 2 
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4 
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0 
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0 1 


Total Re- 
trieved by 

terns (Line 
1) and by 
ASM only 


Judge 1 
Judge 2 
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5 
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0 


4 

0 1 



4. 8.4.2. Derived Performance Characteristics 
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Estimated Mo. of 
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4. 9 Question 9 



4. 9 .1* Question 

4.9 .1.1. Original Statement Additing Nitrogen to Austenitic SS; Effects of N 

4.9 .1.2. Added Notes Methods of additing nitrogen to austenitic stainless 
steels as an alloying agent. Effect of nitrogen on room temperature and 
high temperature mechanical properties, corrosion resistance, weldability 
and formability of austenitic stainless steels. 

4.9 .1.3. UDC Analyst's Notes 

See notes to Question 4 for austenitic stainless steel. The addition of 
: 669. 786 denotes some effect of nitrogen. The third aspects, e.g. 
corrosion resistance can be added by colon (;) though in general, I would 
denote them as the second concept, e.g. 669.15*24*26-194.56: 
621.791.753:6697776 ttie influence of nitrogen on the electric arc welding 
of austenitic stainless steel. 



4. 9 .2. UDC Des rlptors Chosen 




Line Descriptor 


Encoded Descriptor 


Frequency of Posting 


A 669.786 


C669786 


45 


B 669.15... 


C66915$ 


very many 


C -194.56... 


W19456S 


155 
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4. q.4. Results and Analysis 
4. 9.4.1. Relevance Judgements 



PVU'f RPJ, 






Character- 

istic 


Judges 


Total 


Relevant 


Marginally 

Relevant 


Non- 

Re levant 


Retrieved 
by Both 
Systems 


Judge 1 
Judge 2 


0 

0 


0 

0 


0 

0 


0 

0 


Retrieved 
by UDC 
only 


Judge 1 
Judge 2 


2 

2 


2 

1 


0 

1 


0 

0 


Total 
Retrieved 
by UDC 


Judge 1 
Judge 2 


2 

2 


2 

1 


0 

1 


0 

0 


Retrieved 
by ASM 
only^ but 
covered by 
-ISl ABTICS 


Judge 1 
Judge 2 


22 

22 


3 

8 


4 

4 


15 

10 


Total Re- 
trieved by 
both sys- 

terns (Line 
1) and by 
ASM only 


Judge 1 
Judge 2 


22 

22 


3 

8 


4 

4 


15 

10 



4. 9.4.2. Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


Estimated Mo. of 


Precision 


Estimated 

Specificity 


Base 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

■+ 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

+ 

Marginal 


Formula 


B 


B+C 


J 


J +K 


J 


J+K 


JR 


( J+K) (R-tS 


R 


R+S 


B/R 


(b-k:)/(r-»s 


I 


I 


9159 B 


9159(B*-c; 


Judge 1 


n.a. 


n.a. 


n.a. 


n.a. 


2 

I 


2 

I 


n.a. 


n.a. 


Judge 2 


n.a. 


n.a. 


n.a. 


n.a. 


1 

7 


2 

7 


n.a. 


n.a. 








4* 9 .5. Failure analysis 

Type 1: Known relevant documents not retrieved t Tota 1 

Type 2: Non^relevant documents retrieved Total 
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A. 10 Question 10 



A. 20*^* Question 

A. 10.1.1. Original Statement Spheroidization kinetics of pearlite in steels 

A. 10.1.2. Added Notes Spheroidization kinetics of pearlite in carbon and 

alloy steels 



A. 10.1.3. UDC Analyst's Notes 

Spheroidization will come under 621.785.37, processes not involving phase 
transformation. Alloy steels as such is 669.15-194, unless specific 
elements are mentioned, as 669.15*26*28-194 chromium molybdenum steels. 
Kinetics is not easily classified, and I suggest 536.7 thermodynamics, and 
possibly worth a search at 541.124, generalities of chemical dynamics. 



A. 10 
Line 


.2. UDC Descriptors 
Descriptor 


Chosen 

Encoded Descriptor 


Frequency of Posting 


A 


621.785.37 


C62178537 


1 


B 


669 . 15. . . 


C66915$ 


very many 


C 


-194... 


W194$ 


958 


D 


536.7 


C5367 


17 


E 


541.124 


C541124 


42 




A. 10 


.3. Encoded Logical Statements (condensed form) 
(+ = or, * = and, $ = truncation) 




1. 


[A * B ** C ** (D ♦ 


E)] 




2. 


CA$ ** B ** C ** (D$ 


♦ E$)] 





3. (A$ ** B ** C) 







4. 10. A. Results and Analysis 
4.10*^. 1. Relevance Judgements 





Character- 

istic 


Judges 


Total 


Relevant 


Marginally 

Relevant 


Non- 

Relevant 






Retrieved 
by Both 
Systems 


Judge 1 
Judge 2 


0 

0 


0 

0 


0 

0 


0 

0 


Retrieved 
by UDC 
only 


Judge 1 
Judge 2 


0 

0 


0 

0 


0 

0 


0 

0 


Total 
Retrieved 
by UDC 


Judge 1 
Judge 2 


0 

0 


0 

0 


0 

0 


0 

1 

0 


Retrieved 
by ASM 
only, but 
covered by 
'ISI ABTICS 


Judge 1 
Judge 2 


10 

10 


1 

2 


3 

2 


6 

6 


Total Re- 
trieved by 
both sys- 

terns (Line 
1) and by 
ASM only 


Judge 1 
Judge 2 


10 

10 


1 

2 


3 

2 


6 

6 








4.10 Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


Estimated No. of 


Precision 


Estimated 

Specificity 


Base 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

+ 

Marginal 


Formula 


B 


B+C 


J 


J +K 


J 


J + K 


JR 


t J-HO (R-tS 


R 


R +S 


B/R 


(B*»C)/(R-»S 


I 


I 


9159 B 


9159(B<-c; 


Judge 1 


n.a. 


n • d • 


n.a. 


n.a. 


n • & • 


n.a. 


n • 3 • 


* 

n.a. 


Judge 2 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 







4.10 .5. Failure analysis 

Type 1; Known relevant documents not retrieved t Total 
Type 2; Non-relevant documents retrieved Total 



CM 

It 

















4. 11 Question 



4. 11.1. Question 

4. ll.lol. Original Statement Joining Cast Iron to Steel 

4.11.1.2, Added Notes Joining (welding, brazing, soldering, adhesive 
bonuing) of cast iron to steel. 



4. 11.1.3. UDC Analyst's Notes 

All such processes as soldering, brazing and welding are to be found in 
the subdivision of 621.791, e.g. 621. 791. 3S, soft soldering, 621.791.36, 
brazing, 621.791.7S3.9, submerged arc welding. Articles covering more 
than one of the processes will be under 621.791 itself. "Cast Iron to 
Steel" is not an easy concept for any one. 669.131>669.14 is nearest in 
the UDC. 



11 


.2. UDC Descriptors Chosen 




idine 


Descrlotor 


Encoded Descriptor 


Frequency of Postlne 


A 


621.791 


C621791 


74 


B 


621.791... 


C621791S 


3S8 


C 


669.131 


C669131 


16 


D 


669.14 


C66914 


2073 


E 


621.79... 


C62179S 


517 


F 


669. 13. . . 


C66913$ 


369 


G 


669. IS. . . 


C6691S$ 


very many 



4. 11.3. Encoded Logical Statements (condensed form) 







(+ * or, * « and, $ « truncation) 


1. 


(A * 


C * D) 


2. 


(B * 


C$ * D$) 


3. 


CE * 


F * (D$ ♦ G)] 



4JL1 .4. Results and Analysis 
4J1.4.1. Relevance Judgements 





Character- 

istic 


Judges 


Total 


Relevant 


Marginally 

Relevant 


Mon- 

Relevant 






Retrieved 
by Both 
Systems 


Judge 1 
Judge 2 


0 

0 


0 

0 


0 

0 


0 

0 


Retrieved 
by UDC 
only 


Judge 1 
Judge 2 


2 

2 


2 

2 


0 

0 


0 

0 


Total 
Retrieved 
by UDC 


Judge 1 
Judge 2 


2 

2 


2 

2 


0 

0 


0 

0 


Retrieved 
by ASM 
only^ but 
covered by 
'ISI ABTICS 


Judge 1 
Judge 2 


3 

3 


1 

1 


0 

0 


2 

2 


Total Re- 
trieved by 

terns (Line 
1) and by 
ASM only 


Judge 1 
Judge 2 


3 

3 


1 

1 


0 

0 


2 

2 





4,11.4.2. Derived Performance Characteristics 



Charac- 


Estimated 


Estimated Mo. of 


Precision 


Estimated 


teristic 


Recall 








Specificity 




Relevant 


Relevant 


Relevant 


Relevant 


Relevant 


Relevant 


Relevant 


Relevant 

1 


Base 


only 


+ 

Marginal 


only 


'+ 

Marginal 


only 


+ 

Marginal 


only 


+ 

Marginal 


Formula 


B 


B+C 


J 


J +K 


J 


J+K 


JR 


(J-HC^ (R-^S 


R 


R+S 


B/R 


(B-KJ)/(R'»S 


I 


I 


9159 B 


9159(!H-C; 


Judge 1 


11 • 21 • 


n.a. 


n • 21 • 


n.a. 


2 


2 


n.a. 


n • 21 • 












I 


7 






Judge 2 


n • 21 • 


n.a. 


n.a. 


n.a. 


2 


2 


n.a. 


n.a. 












7 


7 




• 

















A. 12 Question 12 






f 



4. 12«1* Question 

4* 12 .1.1. Original Statement Heat Treatment of Steels with Ultrasonics 



4.12*1.2. Added Notes n©at treating any steel using ultrasonics 



4. 12 .1.3. UDC Analyst's Notes 

The heat treatment process should be specified; if not, search should be 
made in 621.785 itself and all its subdivisions, for material additionally 
classified with 534.29*8. This number means "The Effect of Ultrasonic 
Vibrations". 







i 



4. 12 


.2. UDC Descriptors 


Chosen 




Line 


Descriptor 


Encoded Descriptor 


Frequency of Postine 


A 


621. 78S 


C621785 


129 


B 


S34.29 


C53429 


27 


C 


*8 • • • 


W8$ 


64 















4.1^ Results and Analysis 
4.12 *4* 1* Relevance Judgements 



Character- 


Judges 




Total 


Relevant 


Marginally 


Non- 


istic 








Relevant 


Re levant 


Retrieved 
by Both 


Judge 1 




0 


0 


0 


0 


Systems 














Judge 2 




0 


0 


0 


0 


Retrieved 
by UDC 


Judge 1 




2 


2 


0 


0 


only 


Judge 2 


2 


2 


0 


0 




Total 


Judge 1 




2 


2 


0 


0 


Retrieved 






4m 








by UDC 


• 










» 




Judge 2 




2 


2 


0 


0 


Retrieved 












A 


by ASM 


Judge 1 




3 


0 


1 


2 


only^ but 














covered by 
“ISI ABTICS 


Judge 2 


3 


1 


0 


2 


Total Re- 
trieved by 


Judge 1 




3 


0 


1 


2 


both sys- 














terns (Line 
1) and by 
ASM only 


Judge 2 




3 


1 


0 


2 



4.12.4.2. Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


Estimated No. of 


Precision 


Estimated 

Specificity 


Base 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

+ 

Marginal 


Formula 


B 


B + C 


J 


J +K 


J 


J + K 


JR 


(J-40 (R4S: 


R 


R+S 


B/R 


(B-K;)/(R-ts 


I 


I 


9159 B 


9159(B<-C; 


Judge 1 


n.a. 


n.a. 


n.a. 


n.a. 


2 

I 


2 

7 


n.a. 


n.a. 


Judge 2 


n.a. 


n.a. 


n.a. 

f 


n.a. 


2 

7 


2 

7 


n.a. 


n.a. 



m 









mmik 













t- 



-■' \-i>.'i^r- 



■ '. V ■ 






4.13 Question 



4.13.1. Question 



4.13 .1.1. Original Statement Impurity Effect on Mechanical Properties of 

Ship Plate 

4.13.1.2. Added Notes Influence of S, P, 0, C, Cr20,, Al202» fi02 and other 
impurities on all mechanical properties of plates used for ship hulls and 
submarines. 



4.^3 .1.3. UDC Analyst's Notes 

This is another enquiry which is both very general and very specific. As 
in Question 5, the Individual properties if not recoverable at 539.4/. 5 
(when coloned to 669.14.018.293-414, ship building steels, in plate form) 
should be sought separately. 



4. 13 


.2. UDC Descriptors Chosen 




Line 


Descriptor 


Encoded Descriptor 


Frequency of Posting 


A 


669.14 


C66914 


2073 


B 


.018.293.. 


. X18293S 


5 


C 


-414. . . 


W414$ 


34 


D 


539.4 


C5394 


445 


E 


539.5 


C5395 


1 




4.13 


.3. Encoded Logical Statements (condensed form) 






(+ » or. 


* = and, $ B truncation) 




1. 


[A • B • C • 


(D ♦ E)3 




2. 


[A$ • B • C 


* (D$ ♦ E$)3 




3. 


(A$ • B • C) 







o 

-ERIC 







4. 134. Results and Analysis 
413.4.1. Relevance Judgements 



Character- 


Judges 




Total 


Relevant 


Marginally 


Non- 


istlc 














Relevant 


Relevant 


Retrieved ^ 
by Both 


Judge 1 




0 




0 




0 


0 


Systems 


















Judge 2 




0 




0 




0 


0 


Retrieved 
by UDC 


Judge 1 




0 




0 




0 


0 


only 


















Judge 2 




0 




0 




0 


0 


Total 


Judge 1 




0 




0 




0 


0 


Retrieved 
by UDC 




I 




J - 




K 




L 




Judge 2 




0 




0 




0 


0 


Retrieved 






a 








0 




by ASM 


Judge 1 




0 




1 




5 


only^ but 




M 




N - 










covered by 
-I SI ABTICS 


Judge 2 


6 


0 




1 


5 


Total Re- 
trieved by 


Judge 1 




6 




1 




0 


5 


both sys- 




Q 




R - 




S 




T 


terns (Line 
1) and by 
ASM only 


Judge 2 




6 




0 




1 


5 



4.13,4,2. Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


Estimated No. of 


Precision 


Estimated 

Specificity 


Base 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

H- 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

+ 

Marginal 




B 


B + C 


J 


J +K 


J 


J+K 


JR 


(J+Kl (R-tS 


r ormu is 


R 


R +S 


B/R 


(B-K3)/(R+S 


I 


I 


9159 B 


9159(Bt-C; 


Judge 1 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 


H • 21 • 


n.a. 


n.a. 


Judge 2 


n.a. 


n.a. 


n • 21 • 


n.a. 


n.a. 


n.a. 


n.a. 


n • 21 • 




4.14 Question 14 



4.14«1* Question 

4.14.1.1. Original Statement Iron Ore Pellets 

4.14.1.2. Added Notes None. 

4. 14 .1.3. UDC Analyst's Notes 

622.341.1 Iron Ore 

622.341.11 Magnetite 

622.341.12 Haematite 

Add -188.36 for "in the form of pellets" 



4. 14 


.2. UDC Descriptors 


Chosen 




Line 


Descriptor 


Encoded Descriptor 


Frequency of Posting 


A 


622.341.1 


C6223411 


286 


B 


622.341.11 


C62234111 


18 


C 


622.341.12 


C62234112 


11 


D 


622.341... 


0622341$ 


332 


E 


* 188 • 36 • • • 


W18836S 


0* 



•Note: Searches were run according to UDC analyst's notes, without 

knowledge of frequency of postings The search specification precluded 
retrieval of any documents. 



4. 14.3. Encoded Logical Statements (condensed form) 
(+ = or, * * and, $ = truncation) 

1. C(A ♦ B ♦ C) • E) 

2. (D • E) 







4. 144 . Results and Analysis 
4JL4.4.1. Relevance Judgements 



Character- 

istic 


Judges 


Total 


Relevant 


Marginally 

Relevant 


Non- 

Relevant 


Retrieved 

• 

by Both * 
Systems 


Judge 1 
Judge 2 


0 

0 


0 

0 


0 

0 


0 

0 


Retrieved 
by UDC 
only 


Judge 1 
Judge 2 


0 

0 


0 

0 


0 

0 


0 

0 


Total 
Retrieved 
by UDC 


Judge 1 
Judge 2 


0 

0 


0 

0 


0 

0 


0 

0 


Retrieved 
by ASM 
only^ but 
covered by 
-ISI ABTICS 


Judge 1 
Judge 2 


21 

21 


12 

N 

13 


0 

0 

3 


9 

P 

5 


Total Re- 
trieved by 
both sys- 

terns (Line 
1) and by 
ASM only 


Judge 1 
Judge 2 


21 

Q 

21 


12 

13 


0 

s 

3 


9 

T 

5 



4.14*4.2. Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


Estimated No. of 


Precision 


Estimated 

Specificity 


Base 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

H- 

Marginal 


Relevant 

only 

J 0 


Relevant 

Marginal 


Relevant 

only 


Relevant 

+ 

Marginal 


Formula 


B 


B +C 


J 


J +K 


J 


J+K 


JR 


( J+Kl (R+S 


R 


R +S 


B/R 


(B-K3)/(R+S 


I 


I 


9159 B 


9159(B^■C; 


Judge 1 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 


Judge 2 


n.a. 


n.a. 


n.a. 

t 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 
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4. X5 Question 



4*15 Question 

4.15.1.1. Original Statement The Effect of Cr-Ni-Cu-P on Atmospheric 

Corrosion 

4.15.1.2. Added Notes jjig effect of any material, alloying addition or the 
like on atmospheric corrosion, scaling, rusting, oxidation, etc. 

4.15.1.3. UDC Analyst's Notes This is remarkably vague, and the definition 
does not clarify it. Numbers for all the types of corrosion can be found 
at 620.193, e.g. 620.193.2 atmospheric corrosion. 620.193.53 effect of 
combustion gases. 620.193.54 effect of hot oxygen on oxidising gases. 
669.26 Chromium. 669.24 nickel. 669.3 copper. 669.779 phosphorus. 



4 . 15 . 2 . UDC Descriptors 


Chosen 




Line Descriptor 


Encoded Descriptor 


Frequency of Postlne 


A 620.193.2... 


C6201932S 


71 


B 669.26... 


C66926S 


62 


C 669.24... 


C66924$ 


72 


D 669.3... 


C6693$ 


42 


E 669.779... 


C669779$ 


17 



4. 15 .3. Encoded Logical Statements (condensed form) 




(+ ® or, * ® and, $ « truncation) 




1. CA * (B ♦ C ♦ D ♦ E)1 




2. (A) 
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ERIC 







■iiiBi 












la 




4.15.4. Results and Analysis 
4.15.4.1. Relevance Judgements 








Character- 

istic 


Judges 


Total 


Relevant 


Marginally 

Relevant 


Non- 

Relevant 






Retrieved 
by Both 
Systems 


Judge 1 
Judge 2 


1 

1 


0 

1 


1 

0 


0 

0 


Retrieved 
by UDC 
only 


Judge 1 
Judge 2 


E -'•? . - . 

42 


19 

25 


6 

3 


17 

14 


Total 
Retrieved 
by UDC 


Judge 1 
Judge 2 


I . 

43 


19 

26 


7 

3 


17 
’ 14 


Retrieved 
by ASM 
only^ but 
covered by 
-ISI ABTICS 


Judge 1 
Judge 2 


19 

19 


1 

5 


0 

2 


18 

12 


Total Re- 
trieved by 

terns (Line 
1) and by 
ASM only 


Judge 1 
Judge 2 


20 

20 


1 

6 


1 

2 


18 

12 





4.15.4.2. Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


Estimated No. of 


Precision 


Estimated 

Specificity 


Base 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

+ 

Marginal 


Formula 


B 


B'*'C 


J 


J +K 


J 


J + K 


JR 


C J+K) (R-tS 


P. 


R +S 


B/R 


(B-KJ)/(R-*8 


I 


I 


9159 B 


9159(B*-C; 


Judge 1 


0 


1 


19 


26 


19 


26 






1 


2 


WT 


TTT 


TS 


TS 


n • SI • 


.0028 


Judge 2 


1 


1 


26 


29 


26 


29 




.0253 


6 


8 


=1/6 


1/8 


43 


TS 


.0085 
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4. 15.5. Failure analysis 

Type 1: Known relevant documents not retrieved j Total = 

Type 2: Non-relevant documents retrieved Total = 
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l(i Question 



4. Question 

4.i 6*1*1« Original Statement Work Hardening of Steels 



4. Added Notes 



None* 



4. 16. 1.3. UDC Analyst's Notes 



621.787 



A. 16.2. UDC Descriptors 


Chosen 




Line Descriptor 


Encoded Descriptor 


Frequency of Postlne 


A 621.787 


C621787 


13 



A. 16.3. Encoded Logical Statements (condensed form) 
(+ ® or, * = and, $ = truncation) 



1. (A) 

2. (A$) 



o 

UC 















A.16.A. Results and Analysis 
A.16.A.1. Relevance Judgements 



Character- 

istic 



Retrieved 
by Both 
Systems 



Retrieved 
by UDC 
only 



Total 
Retrieved 
by UDC 



Retrieved 
by ASM 
only, but 
covered by 
ISI ABTICS 



Total Re- 
trieved by 
both sys- 



tems (Line 
1) and by 
ASM only 



Judges 



Judge 1 
Judge 2 



Judge 1 
Judge 2 



Judge 1 
Judge 2 



Judge 1 
Judge 2 



Judge 1 
Judge 2 



Total 



0 



0 



0 



E 



0 



0 



0 



10 



M 



10 



10 



10 



Relevant 



0 



B 
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0 



0 



0 



0 



N 



Marginally 

Relevant 



0 



0 



0 



0 



0 



K 



0 



0 



0 



4 



Non- 

Relevant 



0 



0 



0 



H 



0 



0 



0 



4.16.4.2. Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


Estimated No. of 


Precision 


1 Estimated 

Specificity 


Base 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

+ 

Marginal 


Formula 


B 


B + C 


J 


J +K 


J 


J+K 


JR 


(J-HO (R4S 


R 


R+S 


B/R 




I 


I 


9159 B 


9159(B<-C; 


Judge 1 


n.a. 


n.a. 


a SI a 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 


Judge 2 


n.a. 


H a fl • 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 


n.a. 
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A.l7 Question 17 



4.17 .1. 


Question 




4.17 .1.1. 


Original Statement Ultrasonic Testing of Steel Plates 


4.17 .1.2. 


Added Notes 


Detection of Defects in Steel Plates by ultrasonic 
techniques. 


4.17 .1.3. 


UDC Analyst 


's Notes 






669.14-414; 620.179.16 



U. u »2, UDC Descriptors Chosen 



Line 


Descriptor 


Encoded Descriptor 


Frequency of Posting 


A 


669.14 


C66914 


2073 


B 


-414... 


W414$ 


34 


C 


620.179.16 


C62017916 


63 



4. 17.3. Encoded Logical Statements (condensed form) 
(+ = or^ * =? and, $ = truncation) 

1. (A * B • C) 

2. (A$ * B * C$) 

3. (B * C$) 



p 
















4.17.4. Results and Analysis 
4.17.4.1. Relevance Judgements 



Character- 

istic 


Judges 


Total 
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Marginally 

Relevant 


Non- 
Re levant 


Retrieved 
by Both 
Systems 


Judge 1 
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0 
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0 


0 

0 


0 

0 


Retrieved 
by UDC 
only 


Judge 1 
Judge 2 


1 

1 


1 

1 
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0 


0 

0 


Total 
Retrieved 
by UDC 


Judge 1 
Judge 2 
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by ASM 
only^ but 
covered by 
-ISI ABTICS 


Judge 1 
Judge 2 


17 
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Total Re- 
trieved by 
both sys- 

terns (Lire 
1) and by 
ASM only 


Judge 1 
Judge 2 


17 
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9 
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4.17.4.2. Derived Performance Characteristics 
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A. ig Question IR 



4.13 . 1 . Question 

4.1R.1.1. Original Statement Effect of Alloying Elements on the properties 

of low carbon weldable steels. 

4.13.1.2. Added Notes Weldability of carbon steels and effect of alloying 
elements, C, Mn, Si, Cr, Ni, Mo, Cu, Al, Cb, Zr, Ti, Ta, singly or in 
combination. 

4.13 .1.3. UDC Analyst's Notes / 

The alloying elements will be found as in previous examples by their simple 
UDC numbers. Weldability will be found under the precise welding process 
(621.791...). A search under weldability tests, 620.179.2 may provide 
some more information. 



4 . 13 . 2 . UDC Descriptors 
Line Descriptor 

A 669.15... 

B -194... 

C 621.791... 

D 620.179.2... 



Chosen 

Encoded Descriptor 
C66915$ 

W194$ 

C621791$ 

C6201792$ 



Frequency of Posting 
very many 
958 
357 
13 



I 4. 13 , 3 . Encoded Logical Statements (condensed form) 
(+ = or, * = and, $ = truncation) 

1. CA * B (C ♦ D)3 

2. (C ♦ D) 




4.1^. 4. Results and fmalysis 
4.18.4.1. Relevance Judgements 





Character- 

istic 


Judges 


Total 


Relevant 


Marginally 

Relevant 


Non- 
Re levant 




Retrieved 
by Both 
Systems 


Judge 1 
Judge 2 


1 

1 


0 

1 


1 

0 


0 

0 


Retrieved 
by UDC 
only 


Judge 1 
Judge 2 


218 

218 


16 

18 


37 

66 


165 

134 


Total 
Retrieved 
by UDC 


Judge 1 
Judge 2 


219 

219 


16 

19 


38 

66 


165 

134 


Retrieved 
by ASM 
only^ but 
covered by 
-ISI ABTICS 


Judge 1 
Judge 2 


14 

14 


4 

, # 

0 


3 

4 


7 

10 


Total Re- 
trieved by 
both sys- 

terns (Line 
1) and by 
ASM only 


Judge 1 
Judge 2 


15 

15 


4 

1 


4 

4 


7 

10 



4.18.4.2. Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


1 Estimated Ho. of 


Precision 


Estimated 

Specificity 


Base 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

+ 

Marginal 


Formula 


B 


B'*'C 
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J +K 
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J+K 


JR 


( 3 -^) (R-»S 
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R+S 


B/R 


(B-KJ)/(R-»S 
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I 


9159 B 


9159(B^■C; 
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8 
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54 


16 
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n.a. 
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.0472 
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19 Question 19 



4. 19*1* Question 

4.19.1.1. Original Statement Mercury Corrosion of Stainless Steel 

4. 19.1.2. Added Notes Any effect of mercury at ambient temperatures on 

corrosion^ cracking^ embrittlement or any other mechanical properties or 
stainless steels. 

4. 19.1.3. UDC Analyst's Notes 

replies for stainless steel. Embrittlement will be denoted by 
:539.56j Hg is 669.791. For corrosion 620.193.93 (Actually, the effects 
of scorif ication and slagging and thus the only suitable place for molten 
metals) should be searched. 



4. 1!>. 


>2. UDC Descriptors 


Chosen 




Line 


Descrintor 


Encoded Descriptor 


Frequency of Postine 


A 


539.56. . . 


C53956$ 


215 


B 

C 


669.14... 
.018.8. . . 


C66914$ 

X188$ 


very many 
406 


D 


669.791... 


C669791$ 


2 


E 


620.193.93... 


C62019393$ 


7 


F 


620.193... 


C6201935 


355 



4« 19.3. Encoded Logical Statements (condensed form) 
(+ « or, * = and, $ * truncation) 

1. C(A ♦ E) * B * C * D] 

2 . {CB • C • (A ♦ D ♦ E)] ♦ [D • (A + E)fy 

3. (i® * C * (A + D ♦ F)3 ♦ CD » (A ♦ F)]i} 



















449 . 4 . Results and Analysis 
4J-9.4.1. Relevance Judgements 



Character- 


Judges 


Total 


Relevant 


Marginally 


Non- 


istic 






Relevant 


Re levant 


Retrieved 
by Both 


Judge 1 


0 


0 


0 


0 


Systems 


Judge 2 


0 


0 


0 


0 


Retrieved 
by UDC 


Judge 1 


39 


0 


1 


38 


only 












Judge 2 


39 


0 


3 


36 


Total 


Judge 1 


39 


0 


1 


38 


Retrieved 












by UDC 










1 




Judge 2 


39 


0 


3 


36 


Retrieved 




0 


0 


0 


0 


by ASM 


Judge 1 


only^ but 












covered by 
-ISI ABTICS 


Judge 2 


0 


, r 

0 


0 


0 


Total Re- 
trieved by 


Judge 1 


0 


0 


0 


0 














terns (Line 
i) and by 
ASM only 


Judge 2 


0 


0 


0 


0 



4.19.4.2. Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


Estimated No. of 


Precision 


Estimated 

Specificity 


Base 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

+ 

Marginal 


Formula 


B 


B ■^'C 


J 


J +K 


J 


J + K 


JR 


CJ-tR) (R+s: 


R 


R+S 


B/R 


(B-KJ)/(R-»S 


I 


I 


9159 B 


9159(B^C; 


Judge 1 


n.a. 


n.a. 


n.a. 


n.a. 


0 


1 

Tgr 


n.a. 


n.a. 


Judge 2 


n • 21 • 


n • 21 • 


n.a. 


n.a. 


0 

77 


3 

77 


n • 21 • 


n.a. 



r 






4vt?swr>K:Kr»»sB^ 









4. 20 Question 20 



4.20*1* Question 

4.20.1.1. Original Statement Improvement of Mechanical Properties of Steel 

by Combined Deformation and heat treatment 

4.i20.i.2. Added Notes Mechanical Properties of Steels as influenced by 
Ausforning, Ausworking, Ausrolling or by Ausforging. 



4.20.1.3. UDC Analyst's Notes 

Not an easy one. 621.785+621.787 (Equals heat treatment plus work 
hardening) 



4. 20.2. UDC Descriptors Chosen 



Line Descriptor 
A 621.785 

B 621.787 



Encoded Descriptor 

C621785 

C621787 



Frequency of Posting 
129 
13 



4. 20.3. Encoded Logical Statements (condensed form) 
(+ * or^ * = and, $ ® truncation) 

1. (A * B) 

2. (A$ * B$) 







4.20.4. Results and Analysis 
4.20.4.1. Relevance Judgements 









Character- 
istic 



Retrieved 
by Both 
Systems 



Judges 



Judge 1 



Judge 2 



Total 



Relevant 



0 



B 



0 



Marginally 

Relevant 



0 



0 



Non- 

Re levant 



0 



0 



Retrieved 
by UDC 
only 



Judge 1 



E 



Judge 2 



0 



0 



0 



H 



0 



Total 
Retrieved 
by UDC 



Judge 1 



Judge 2 



0 



K 



0 



Retrieved 
by ASM 
only^ but 
covered by 
-ISI ABTICS 



Judge 1 
Judge 2 



8 



M 



N 



8 



Total Re- 
trieved by 
both sys- 
tems (Line 
1) and by 
ASM only 



Judge 1 
Judge 2 



8 



8 



1 

1 



0 



0 



0 



0 



4.20.4.2. Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


Estimated No. of 


Precision 


Estimated 

Specificity 


Base 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

+ 

Marginal 


Formula 


B 


B+C 


J 


J +K 


J 


J+K 


JR 


(J+K) (R+S 


R 


R+S 


B/R 


(B-K3)/(R-*8 


I 


I 


9159 B 


9159(B+C: 


Judge 1 


n.a. 


n.a. 


n.a. 


n.a. 


2 

7 


2 

I 


n.a. 


n.a. 


Judge 2 


n«a. 


n.a. 


n.a. 


n.a. 


2 

2 


2 

7 


n.a. 


n.a. 
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4. 21 Question 21 



A. 21* 1* Question 

4. 21*1«1* Original Statement Internal Friction of Fe Alloys and Co and Ni 

Superalloys 

4. 21 * 1*2. Added Notes Programned for internal friction in iron-base alloys 
including steels only. Includes Co-base and Ni-base Superalloys only when 
have Fe as a major component. 

4. 21.1*3. UDC Analyst's Notes 

1) 669.15: Iron Alloys 

539.67: Internal Friction 

2) 669.15*24: Iron-Nickel Alloys 

539.67 

3) 669.15*25: Iron-Cobalt Alloys 

539.67 

4) 669.15*24*25: Iron-Nickel Cobalt... 

539.67 



4. 21.2. UDC Descriptors Chosen 




Line Descriptor 


Encoded Descriptor 


Frequency of Posting 


A 669.15 


C66915 


411 


B 539.67... 


C53967$ 


27 


C 669.15*24 


C66915Y24 


139 


D 669.15*25 


C66915Y25 


11 


E 669.15*24*25 


C66915Y24Y25 


0 



4. 21*3* Encoded Logical Statements (condensed form) 
(+ = or> * ~ and, $ *= truncation) 




















4.21.4. Results and Analysis 
4.21.4^1. Relevance Judgements 
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by Both 
Systems 
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only 


Judge 1 
Judge 2 
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16 
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by ASM 
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covered by 
-ISI ABTICS 


Judge 1 
Judge 2 
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trieved by 
both sys- 
tems (Line 
1) and by 
ASM only 
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Judge 2 


10 
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4. 214.2. Derived Performance Characteristics 
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A. 22 Question 22 



A*22*^« Question 

4.22.1.1. Original Statement Continuous Casting of Stainless and Alloy Steels 
4.22 .1.2. Added Notes 

None. 

4.22 .1.3. UDC Analyst's Notes 

1) 669.14.018.8: Stainless StOels 

621.74.047 Continuous Casting 

2) Iron, Fe-Ni, Fe-Co , Fe-Ni-Co alloys as above. 



4.22 


.2. UDC Descriptors 


Chosen 




Line 


Descriptor 


Encoded Descriptor 


Frequency of Posting 


A 


621.74 


C62174 


133 


B 


.047... 


X47$ 


125 


C 


669.14 


C66914 


2073 


D 


.018.8... 


X188$ 


406 


E 


669.15 


C66915 


411 



4.22.3. Encoded Logical Statements (condensed form) 
(+ « or, * ® and, $ * truncation) 

1. CA * B * D * (C ♦ E)1 

2. CA$ * B * D * (C$ ♦ E$)l 

3. (A$ * B) 
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A. 22A. Results and Analysis 
A. 22 A.I. Relevance Judgements 



Character- 

istic 


Judges 


Total 


Relevant 


Marginally 

Relevant 


Non- 

Relevant 


Retrieved 
by Both 
Systems 


Judge 1 
Judge 2 


9 

9 


4 

3 


2 

6 


3 

0 


Retrieved 
by UDC 
only 


Judge 1 
Judge 2 


53 

53 


12 

6 


13 

45 


28 

2 


Total 
Retrieved 
by UDC 


Judge 1 
Judge 2 


62 

I 

62 


16 

J 

9 


i 

15 

51 


31 

L 

» 

2 


Retrieved 
by ASM 
only^ but 
covered by 
-ISI ABTICS 


Judge 1 
Judge 2 


8 

M 

8 


5 

N - 

b 


1 

0 

6 


2 

P 

2 


Total Re- 
trieved by 
both sys- 
tems (Line 
1) and by 
ASM only 


Judge 1 
Judge 2 


17 

Q 

17 


9 

R 

3 


3 

S 

12 


5 

T 

2 



i 

\ 

\ 



A.22*A.2. Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


Estimated No. of 


Precision 


Estimated 

Specificity 


Base 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 

Marginal 
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only 


Relevant 

+ 

Marginal 


Formula 
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B+C 
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J +K 
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JR 


(J-40 (R-tS 
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R +S 


B/R 
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I 
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9159 B 
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Judge 1 


4 
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TI 


16 
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31 
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31 

FI 


.0039 


.0068 


Judge 2 
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15 
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60 

97H 
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60 

FI 


.0010 


.0109 
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4. 23 Question 23 



A* 23.1. Question 

A. 23.1.1. Original Statement High Temperature Extrusion of Steels and 

Apparatus Therefor 

A. 23.1.2. Added Notes 

None. 



A. 23.1.3. UDC Analyst's Notes 

621.777.2.016.2 covers both aspects of the definition. 



1 . 

2 . 

3. 

4. 



(A * B) 

(A$ * B) 

C(A$ ♦ C) * B] 
(A$ ♦ C) 



er|c 



A. 23.2. UDC Descriptors 


Chosen 




Line Descriptor 


Encoded Descriptor 


Frequency of Posting 


A 621.777.2 


C6217772 


16 


B .016 . 2 . . . 


X162$ 


87 


C 621.774.38... 


062177438$ 


6 




A. 23.3. Encoded Logical Statements (condensed form) 
(+ = or^ * = and, $ = truncation) 






i 
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4.23.4. Results and Analysis 
4.23.4.1. Relevance Judgements 



Character- 


Judges 




Total 


Relevant 


Marginally 


Non- 


istic 










Relevant 


Re levant 


Retrieved 
by Both 


Judge 1 




3 




1 


0 


2 


Systems 


Judge 2 




3 


1 


0 


2 




Retrieved 
by UDC 


Judge 1 




22 




18 
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4 


only 


Judge 2 




22.. 
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11 
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Total 


Judge 1 




25 




19 
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Retrieved 
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f 




Judge 2. 




25 
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11 
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Retrieved 
by ASM 


Judge 1 




15 
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3 


only, but 
















covered by 
-ISI ABTICS 


Judge 2 


15 


3 


3 


9 


Total Re- 
trieved by 


Judge 1 




18 




8 


5 


5 


















terns (Line 
1) and by 
ASM only 


Judge 2 




18 




4 


3 


11 



t 




4.23.4.2. Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


Estimated No. of 


Precision 


Estimated 

Specificity 


Base 


Relevant 

only 
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Marginal 
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Formula 
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8 1 
TT 1 
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.0035 


.0145 





iHliiii 



iiMi 







iiaittiiaiaiftiiiiiiiiiiiaaai^^ 




4.23 .5. Failure analysis 

Type 1: Knovm relevant documents not retrieved [Total 

Type 2: Non-re levant documents retrieved Total 












vO 





















o 








u 
















4J 








0> 




• 










<u 




g 






o -o 




Sf 










JS 


3 


o 






4J CO 




• 






• 




4J 


<u 


u 






o 




CO 






0> 






M 


CM 






>% u 




a 






u 




c 


M 








^ -o 




CO 






3 




•H 


<u 


3 






e 




• 












CM 


<U 






G ca 




CO 






•H 




4J 


<U 


a 










CM 






CO 




CO 


M 


a 






•o G 




• 






CM 




•H 




o 






G > 




St 










X 


CO 


u 






X o> 














<U 


4J 


3 






(U I-I 




>i 






d 






d 








TJ 




•o 






•H 




4J 


9i 


d 






6 ^ 










X 




o 


g 


o> 






•H CO 




•3 


• 




o 




c 


3 


0> 






O 




<U 


4J 




'Q 






O 


.o 






4J 1-1 




> 


c 




ti 




CO 


o 








d ^ 




<u 


Si 




M 




<u 


3 


0> 






.O O 




*H 


B 








o 




> 






u 




U 


<u 










<U 


CO 






4J CO 




4J 


dO 




• 






CO 


JS 






O U 




<u 






0> 




cn 


o; 








CO <U 




M 


3 








• 


JS 


4J 






M *H 






•r-l 




4J 






4J 


o 






4-» JS 




e 






•H 




St 




d 






(0 




<u 






4J 




• 


CM 








.o o 




<u 


€0 








St 


O 


3 






(0 15 




.o 


•H 




d 




CO 










4J 






4J 




•H 




cn 




3 






6 




<u 


d 










rH 


O 






•H ^ 




> 


<u 




4J 




• 


<4 


JS 






CM 




CO 


M 




•H 




• 3 




CO 






4J • 




JS 


(U 




O 




ca G 










G vO 










•H 




CO *Q 


• 


CM 






<U p-l 




TJ 


c 




rH 




G d 


0> 


• 






ca O 






•H 




a 




U G 


CO 


vO 






<u • 




3 






X 




O 4J 


o 


rH 






M 




o 






<u 




u c 


o 


o 






a • 




JS 


<U 








(X «H 




• 










CO 






4J 














^ -s CM 






rO 




3 




00 CO 


o 


>i 






CM vO 






!S 




.O 




d *H 


o 


u 






• 




• 


d 








•H 


4J 


CO 






vO U 




M 


o 








X lo 




•H 






p-i O 




O 


•H 








o> 


CO 


rH 






O 4J • 






4J 




<U 




3 ^ 


CO 


•H 






• a G 




M 


CO 




X 




C 




X 






*H M 




<U 


<U 




<u 




•H • 




3 






M 3 






3 




•o 




OS 


St 


CO 






o 




JS 






e 




G CO 


• 








• CO *H 




o 






•H 




jz cn 


CO 


<U 






rH (D CO 




u 










M 


• 








CM -G «W 




CO 


• 




4J 




1 


CO 


H 






vO 




<u 


u 




O 




d 


CM 








w 0) do 




CO 


<u 




d 




iH U 


• 








•C 6 






a 








P 


St 


• 






r 4J *H 


• 


0) 


CO 








u 5 




CO 






C X 


rH 


d 


a 




c 




o 


d 


d 






O ds <U 


• 


•H 






o 




U CM 


o 


o 






•H «D 


rH 


X 






•H 




u o 


•H 


•H 






(0 a 


• 


O 


<u 




CO 




<u 


4J 


4J 




<u 


3 TJ M 


m 


S 


•H 




3 




d 


CO 


•H 




Lj 


M <U 


• 


g 


> 




U 




d o 




3 




a 


4J *H 


CO 




Q> 




4J 




O *H 


3 


d 






X • 


CM 


TJ 


U 




X 




•H 4J 


i 


o 




*p4 


<u aT) 


• 


CD 






Q> 




M *H 


c 


o 




m 


: 6 <u 


<t 


d 


rH 








P 3 


o 






£ 


•H *H 




•H 


CO 








•H 0) 


CM 


dO 




rM 


4J M-l 


(0 


CO 


M 




4J 




M ^ 




d • 




■ « 


a 4J *H 


3 




0) 




a 




O >% 


x: 


.rH 43 






(D C O 




a 


d 




0) 




CO dO 


o 


rX O 




11 * 


O <U (D 


<U 


X 


<u 




o 




d M 


u 


U U 






6 4J a 


g 


(U 


do 




d 




CO 3 


CO 


O CO 






O X (0 


CO 


d 






o 




u ^ 


0) 


^ <U 




§ 


C? (U 


CO 




<4 




u 




H •-* 


cn 


CO 




W 


c 














CO 




3 




(0 


<U CO 














4J 




^ 0) 




CO 

ill 


• JS JS 


• 


• 


• 




• 




• 0) 


• 


O 4S 




Ql 

Pi 


CO 4J 4J 


3 


CO 


CO 




CO 




*£> S 


CO 


O 4J 


















CO 
























Q> 
























•H 
























4J 












<u 










o\ 


U 


X 










M 












<u 


0) 










3 6 










• 












4J CO 










m 


O 


cx 










CO 










00 


u 


g 










u ^ 


CO 






0) 




cx 


o 










<U dO 


M 






M 


• 




o 










a 6 


0) 






3 
















CM g *H (0 


•C 






4J 


CM 


CO 


M 










• 5 g f-H 


4J 






O 


vO 


o 


O 










vO ^ M CO 


o 






CO 


•• 


•H 












^ O »H 








CM 


cn 


CO 


3 










O 4S M-4 M 


'O 






3 


• 


ds 


<U 










• dO (U 


d 






d 






d 










1^ »H 4J 


CO 






A 


St 


cx 


•H 










• X dO 3 








g 


• 




•g 










fH 3 g 


CO 


CO 






St 




i ^ 










CM •• «H 


• 






0) 


CO 


• • 


o d 










vO Cd »X CM 




O 




a 


cn 


CO 


O 0) 










•• f-H M O 


CM 


• 




•H 


•• 




g 










<r 0 ^ o 


• 


00 




a 


St 


(U 


•• 4J 








OG 


0\ <U ^ 4J 


vO 


CO 






as 


0) 


p-1 CO 








6 


•-I 4J C 


rH 


• 




"O 


rH 


4J 


Q> 








•H 


1 CO O 0) 


o 


St 


St 


d 


1 


CO 


• M 








X 


m »H g 


• 






CO 


cn 




.O 4J 








<u 


•M ds 4J M 










iH 


>1 










'S 


• O CO 3 


• 


• 


• 


0) 


• 


o 


<U 4J 








c 


0% i-l CO <U 






rH 


•O 


o\ 




0) CO 








M 


vO ^ ^ M 


CM 


CM 


CM 


3 


vO 


rH 


CO (U 










vO <! (X 4J 


vO 


vO 


vO 


H 


vO 


<1 










o‘ 


















vO 










CM 


CO 


St 




cn 






1 

rH 






0> 
























M< 

>^ 












rH 






CM 






H 





























irnmmimmmmmmmm iiaiiia&aiaiiiafeaiaiia^^ 









4.24 Question 24 



4.24 •!• Question 

4.24.1.1. Original Statement Vacuum Metallurgy of Steels 



4.24.1.2. Added Notes None. 



4.24 .1.3. TIDC Analyst's Notes 

In general 669-982 will recover all on degassing or any other operation 
on Liquid steel. Vacuum Heat treatment requires the addition of .061 to 
the number for the process. 



4.24 


.2. UDC Descriptors 


Chosen 




Line 


Descriptor 


Encoded Descriptor 


Frequency of Posting 


A. 


669. . . 


C669$ 


very many 


B 


-982... 


W982$ 


184 


C 


62. . . 


C62$ 


very many 


D 


.061. . . 


X61$ 


11 






A.24.A. Results and Analysis 
A.24.A. 1. Relevance Judgements 



Character- 

istic 


Judges 


Total 


Relevant 


Marginally 

Relevant 


Non- 

Re levant 


Retrieved 
by Both 
Systems 


Judge 1 
Judge 2 


1 

1 


1 

1 


0 

0 
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0 


Retrieved 
by UDC 
only 


Judge 1 
Judge 2 


182 

182 


140 

157 


36 

22 


6 

3 


Total 
Retrieved 
by UDC 


Judge 1 
Judge 2 


183 

183 


141 

158 


36 

22 


6 

3 


Retrieved 
by ASM 
only^ but 
covered by 
‘ISI ABTICS 


Judge 1 
Judge 2 


4 

4 


2 

0 


0 

2 


2 

2 


Total Re- 
trieved by 
both sys- 

terns (Line 
1) and by 
ASM only 


Judge 1 
Judge 2 


5 

5- 


3 

1 


0 

2 


2 

2 



A. 24. A. 2. Derived Performance Characteristics 



Charac- 

teristic 


Estimated 

Recall 


Estimated No. of 

5S^fS!""AfiK5!r"“ 


Precision 


Estimated 

Specificity 


Base 


Relevant 

only 


Relevant 

+ 

Marginal 


Relevant 

only 


Relevant 

*»■ 

Margins 1 


Relevant 

only 


Relevant 

Marginal 


Relevant 

only 


Relevant 
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Marginal 


Formula 
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B+C 
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J +K 
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JR 
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B/R 


(b-k:)/(r+s 
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9159 B 
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TST 
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.0461 


.0580 


Judge 2 
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183 


ITS 


.0172 
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.0590 
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ERIC 

mmm 






MM 



liKlifiiii 









4.24 .5. Failure analysis 

Type 1: Known relevant documents not retrieved I Total 

Tvne 2: Non-relevant documents retrieved Total 
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4.25 Question 25 



4 .‘^ 25 .i. Question 

4.25.1.1. Original Statement Vanadium in Steels 

A 25 1 2 Added Notes Influence of Vanadium additions to steels on quenching» 
M^eiing/hardS. phase composition, carbide structures and mechanical 

properties. 

4. 25*I*3* UDC Analyst's Notes 

In fteneral 669.292 will recover all on the effects of Vanadium in steels. 
I? 5i?? the recover material in determination of vanadium and the very 

little on vanadium itself. 
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A.25*^* Results and Analysis 
4.25*^*1* Relevance Judgements 















Character- 
istic 
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Total 



Relevant 



Marginally 

Relevant 



Non- 

Relcvant 



Retrieved 
by Both 
Systems 



Judge 1 
Judge 2 



B 



0 



0 



0 



Retrieved 
by UDC 
only 



Judge 1 
Judge 2 



18 



10 



E 



F 



18 



17 



H 



8 



Total 
Retrieved 
by UDC 



Judge 1 



18 



10 



Judge 2 



18 



17 



K 



8 



Retrieved 
by ASM 
only^ but 
covered by 
‘ISI ABTICS 



Judge 1 
Judge 2 
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4 



Total Re- 
trieved by 
both sys- 
tems (Line 
1) and by 
ASM only 



Judge 1 
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4 
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0 



4.25.4.2. Derived Performance Characteristics 



Charac- 
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4.25 .5. Failure analysis 

Type 1: Known relevant documents not retrieved Total 

Type 2: Non-relevant documents retrieved Total 
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